
y-g - 

1 


1 J 1 







( 



*•, -tjF' MjrJi 




1 * 

n c 








k v Jn\ 

■ ■ j 

W ' \ 


r / . 


MOBILIZING 
VOLUNTEER TUTORS 
TO IMPROVE 
STUDENT LITERACY 


mdrc 

BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 


Robin Tepper Jacob 
Catherine Armstrong 
Jacklyn Altuna Willard 


Implementation, Impacts, 
and Costs of the 
Reading Partners Program 


March 2015 



MDRC BOARD OF DIRECTORS 


Mary Jo Bane 

Chair 

Thornton Bradshaw Professor of 
Public Policy and Management 
John F. Kennedy School of 
Government 
Harvard University 

Robert Solow 

Chairman Emeritus 
Institute Professor Emeritus 
Massachusetts Institute of 
Technology 

Rudolph G. Penner 

Treasurer 

Senior Fellow and Arjay and 
Frances Miller Chair in 
Public Policy 
Urban Institute 


Robert E. Denham 

Partner 

Munger, Tolies &, Olson LLP 

Ron Haskins 

Senior Fellow, Economic Studies 
Co-Director, Center on Children and 
Families 

Brookings Institution 

James H. Johnson, Jr. 

William Rand Kenan Jr. 
Distinguished Professor of 
Strategy and Entrepreneurship 
Director, Urban Investment 
Strategies Center 
University of North Carolina 

Lawrence F. Katz 

Elisabeth Allison Professor of 
Economics 
Harvard University 

Bridget Terry Long 

Professor of Education and 
Economics 

Graduate School of Education 
Harvard University 


Josh B. McGee 

Vice President of Public 
Accountability 

Laura and John Arnold Foundation 

Richard J. Murnane 

Thompson Professor of Education 
and Society 

Graduate School of Education 
Harvard University 

Jan Nicholson 

President 

The Grable Foundation 

John S. Reed 

Retired Chairman 
Citigroup 

Michael Roster 

Former General Counsel 
Stanford University 
Former Managing Partner 
Morrison &. Foerster, Los Angeles 

Cecilia E. Rouse 

Dean, Woodrow Wilson School of 
Public and International Affairs 
Katzman-Ernst Professor in the 
Economics of Education 
Professor of Economics and 
Public Affairs 
Princeton University 

Isabel V. Sawhill 

Senior Fellow, Economic Studies 
Co-Director, Center on Children and 
Families 

Brookings Institution 


Gordon L. Berlin 

President, MDRC 


Mobilizing Volunteer Tutors 
to Improve Student Literacy 


Implementation, Impacts, and Costs 
of the Reading Partners Program 


Robin Tepper Jacob 
(Institute for Social Research, 
University of Michigan) 

Catherine Armstrong 
Jacklyn Altuna Willard 
(MDRC) 

with 


A. Brooks Bowden 
Yilin Pan 

(Center for Benefit-Cost Studies of Education, 
Teachers College, Columbia University) 


March 2015 


mdrc 

BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 



Funders for the Reading Partners Evaluation 


This report is based on work supported by the Social Innovation Fund (S1F), a key White Flouse 
initiative and program of the Corporation for National and Community Service (CNCS). The Social 
Innovation Fund combines public and private resources with the goal of increasing the impact of 
innovative, community-based solutions that have compelling evidence of improving the lives of 
people in low-income communities throughout the United States. 

The Edna McConnell Clark Foundation is leading a S1F project that includes support from CNCS and 
15 private co-investors: The Edna McConnell Clark Foundation, The An nie E. Casey Foundation, The 
Duke Endowment, The William and Flora Flewlett Foundation, The JPB Foundation, George Kaiser 
Family Foundation, The Kresge Foundation, Open Society Foundations, The Penzance Foundation, 
The Samberg Family Foundation, The Charles and Lynn Schusterman Family Foundation, The Starr 
Foundation, Tipping Point Community, The Wallace Foundation, and the Weingart Foundation. 

Dissemination of MDRC publications is supported by the following funders that help finance 
MDRC’s public policy outreach and expanding efforts to communicate the results and implications of 
our work to policymakers, practitioners, and others: The An nie E. Casey Foundation, The Flarry and 
Jeanette Weinberg Foundation, Inc., The Kresge Foundation, Laura and John Arnold Foundation, 
Sandler Foundation, and The Starr Foundation. 

In addition, earnings from the MDRC Endowment help sustain our dissemination efforts. Contributors 
to the MDRC Endowment include Alcoa Foundation, The Ambrose Monell Foundation, Anheuser- 
Busch Foundation, Bristol-Myers Squibb Foundation, Charles Stewart Mott Foundation, Ford 
Foundation, The George Gund Foundation, The Grable Foundation, The Lizabeth and Frank Newman 
Charitable Foundation, The New York Times Company Foundation, Jan Nicholson, Paul H. O’Neill 
Charitable Foundation, John S. Reed, Sandler Foundation, and The Stupski Family Fund, as well as 
other individual contributors. 


The findings and conclusions in this report do not necessarily represent the official positions or 
policies of the funders. 

Cover photos courtesy of Reading Partners. 


For information about MDRC and copies of our publications, see our website: www.mdrc.org. 
Copyright © 2015 by MDRC®. All rights reserved. 



Overview 


Thi s study reports on an evaluation of the Reading Partners program, which uses community volun- 
teers to provide one-on-one tutoring to struggling readers in underresourced elementary schools. 
Established in 1999 in East Menlo Park, California, Reading Partners’ mission is to help children 
become lifelong readers by empowering communities to provide individualized instruction with 
measurable results. At each school, Reading Partners transforms a dedicated space into a “reading 
center,” places a full-time team member on site to manage day-to-day operations, and recruits a 
corps of 40 to 100 community volunteers to work one-on-one with students in pull-out sessions dur- 
ing the school day or after school in kindergarten through grade 5. (This evaluation included only 
students in grades 2 through 5.) 

In March 2011, Reading Partners received a three-year True North Fund investment of up to $3.5 
million in grants from the Edna McConnell Clark Foundation and the Social Innovation Fund, 
matched by $3.5 million from True North Fund co-investors, to further expand its early-intervention 
literacy program to elementary schools throughout the country and evaluate its effectiveness. This 
report is the second publication from that evaluation. A policy brief released by MDRC in June 2014 
reported the initial findings from the evaluation, which was conducted during the 2012-2013 school 
year in 19 schools in three states, with more than 1,100 students randomly assigned to the study’s 
program and control groups. 

This report builds on those initial findings by describing the Reading Partners program and its im- 
plementation in greater detail, exploring whether the program is more or less effective for particular 
subgroups of students, and assessing some of the potential explanations for the program’s success to 
date. In addition, this report includes an analysis of the cost of implementing the Reading Partners 
program in 6 of the 19 sites. 

Key Findings 

• Despite the myriad difficulties inherent in operating a program whose direct service providers 
are volunteers, Reading Partners was implemented in the schools with a relatively high degree 
of fidelity to the program model. On average, students in the study received approximately 1.5 
tutoring sessions per week, and spent 28 weeks in the Reading Partners program. 

• Reading Partners had a positive and statistically significant impact on three different measures 
of student reading proficiency. These impacts are equivalent to approximately one and a half to 
two months of additional growth in reading proficiency among the program group relative to the 
control group and are robust across a range of student characteristic subgroups as well as across 
groups of students who had different levels of reading comprehension skills at the start of the 
study. 

• Reading Partners is a low-cost option for underresourced schools because a majority of the costs 
are in-kind contributions, primarily from community volunteers. On average, schools bear only 
about 20 percent ($710 per program group student) of the total cost of the resources required to 
implement the program, and over half of these costs are in-kind contributions of space and staff 
time from the school. 
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Preface 


Over the last two decades, numerous federal, state, and local efforts have focused on improving 
the literacy skills of America’s young people. Yet, despite these efforts, only limited progress 
has been made. One approach that has consistently shown promise in improving literacy out- 
comes, especially for young children, is one-on-one tutoring. One-on-one tutoring delivered by 
certified teachers has repeatedly demonstrated large positive impacts on the reading proficiency 
of struggling readers. Yet, while this approach has a solid research base demonstrating its effec- 
tiveness, it is both time- and resource-intensive, placing a heavy burden on teachers and schools, 
and thus is an expensive way to ameliorate the problem of low literacy. As such, it may not al- 
ways be a viable option for already underresourced schools. 

This report explores another model: using community volunteers to provide tutoring to 
struggling readers, but in a structured, programmatic framework designed and managed by a 
dedicated nonprofit organization. The Reading Partners program recruits community volunteers 
who devote a few hours each week to tutoring students in kindergarten through grade 5 in read- 
ing, using a structured curriculum. Tutors come from varied backgrounds, are not required to 
have experience working with children or teaching reading, and receive only limited training 
before beginning tutoring. But children are assessed and tutors use specific materials supplied 
by the program, while a site coordinator ensures that each student receives the intended instruc- 
tion, advises tutors whose students have specific difficulties, and fills in when tutors are unable 
to make appointments. 

This evaluation reaches the encouraging conclusion that the Reading Partners program 
successfully improved students’ reading proficiency, even among children in the upper elemen- 
tary grades. Furthermore, the cost to the schools was quite low and substantially less than the 
costs of other supplemental reading services that are typically offered to struggling readers. All 
this suggests that strong volunteer tutoring programs, like Reading Partners, may be a cost- 
effective option for underresourced schools and deserve greater attention in the national effort to 
improve literacy skills. 


Gordon L. Berlin 
President, MDRC 
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Executive Summary 


The Reading Partners program uses community volunteers to provide one-on-one tutoring to 
struggling readers in underresourced elementary schools. Established in 1999 in East Menlo 
Park, California, Reading Partners is a not-for-profit corporation whose mission is to help chil- 
dren become lifelong readers by empowering communities to provide individualized instruction 
with measurable results. The Reading Partners model is based on the premise that too many 
children in low-income communities are not reading proficiently and that many teachers, 
schools, and parents in those communities lack the resources and infrastructure to address the 
problem. 

The evaluation that is described in this report finds that the Reading Partners program 
successfully improved students’ reading comprehension, sight word efficiency, and fluency 
over the course of the school year by an amount that is roughly equivalent to one and a half to 
two months of learning. In addition to demonstrating these measurable impacts, the evaluation 
provides evidence that the cost to the schools was less than half the costs to schools of other 
supplemental reading programs. The costs for Reading Partners were lower because the volun- 
teer tutors accounted for a large share of the resources that were used. Thus, this study provides 
additional evidence that volunteer programs can work and that one-on-one tutoring is effective 
in improving academic outcomes. Furthermore, the results suggest that effective tutoring pro- 
grams, like Reading Partners, may be a cost-effective option for underresourced schools, 
because they bring additional resources to the school through community volunteers. 


Background 

In March 2011, Reading Partners received a three-year True North Fund investment of up to 
$3.5 million in grants from the Edna McConnell Clark Foundation and the Social Innovation 
Fund, matched by $3.5 million from True North Fund co-investors, to further expand its 
early-intervention literacy program to elementary schools throughout the country and evaluate 
its effectiveness. 

In addition to answering questions about the effectiveness of the Reading Partners 
program, the evaluation was designed to examine the potential for volunteer tutoring more 
generally to help improve the reading proficiency of struggling readers. To meet its design 
objectives, the evaluation included an implementation study, an impact study, and a cost 
study. The implementation and impact studies included 19 schools with Reading Partners 
sites in three different states, and the cost study included a subsample of six of these schools, 
also across three different states. Together, the three facets of the evaluation are designed to 
address the following broad sets of research questions: 
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1. In what context was the Reading Partners program implemented, and was it 
implemented as intended — that is, with fidelity to the model? How much 
variability in fidelity of implementation was observed across the sites? What 
factors contributed to any observed variability? 

2. On average, did the Reading Partners program have a positive impact on stu- 
dents’ reading proficiency across three key components of early reading abil- 
ity: sight word efficiency, reading fluency, and comprehension? 

3. What resources are needed to implement the Reading Partners program as 
described in this evaluation and what proportion of the costs of implementing 
the program are borne by the school? 

The implementation study included site visits to all the schools participating in the 
evaluation in the winter of the study year, interviews with key program and school staff and 
volunteers, and the collection of programmatic data from the sites and from Reading Partners’ 
own management information system (MIS). The impact study used a student-level randomized 
controlled trial design, in which students were randomly assigned within each school either to a 
program group that would participate in Reading Partners during the 2012-2013 school year or 
to an “as is” control condition. A total of 1,265 students in grades 2 through 5 across the 19 
schools were randomly assigned. Students were assessed on three different measures of reading 
proficiency in the fall and spring of the study year. Finally, the cost study calculated the inter- 
vention’s total cost by summing the costs of all the resources that were necessary to implement 
the program. 


The Reading Partners Model 

Reading Partners is a “pull-out” program (meaning that students are pulled out of their regular 
classrooms or after-school program for a limited time period in order to meet with their tutors) 
that operates both during the school day and after school in designated spaces called “reading 
centers” at each of the partner schools. The program aims to serve students in kindergarten 
through grade 5 who are half a year to two and a half years behind grade level in reading, who 
are conversationally fluent in English, and who do not have any special needs (that is, do not 
have an Individualized Education Program). 1 The Reading Partners program consists of twice- 
weekly, one-on-one tutoring sessions that last 45 minutes each. Students are tutored by commu- 
nity volunteers who need not have any experience working in education or with children. Tutor- 


1 Individualized Education Programs are developed for children who are found through assessment to have 
a disability that affects their learning process. The program outlines how teachers will help these students learn 
more effectively, considering each of their learning styles and needs. 
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ing sessions are overseen by site coordinators — full-time Reading Partners team members, 
who are usually AmeriCorps members and who are also responsible for managing the day-to- 
day operations of the program . 2 

The Reading Partners model consists of six core components: 

• Regular, one-on-one tutoring 

A key feature of the Reading Partners model is to provide students with individualized 
reading instruction. This one-on-one support — as opposed to small-group support — is the 
main component that differentiates Reading Partners from many other supplemental services 
that are available to struggling readers. 

• Dedicated school space and use of materials 

Reading Partners tutoring is designed to take place on school grounds in a dedicated 
space, or reading center, which contains specific features designated by Reading Partners, such 
as work stations for tutor-student pairs, a library with materials that are suitable for readers at 
different levels, an area with resources for volunteers to use, and posters and other materials 
with a reading theme that can be displayed on the wall. 

• Structured and individualized curriculum 

Tutoring sessions follow a consistent structure. The tutor begins by reading aloud to 
model fluent reading and pausing periodically to ask the student comprehension questions or to 
check the student’s knowledge of key vocabulary. The tutor then uses curricular materials to 
introduce or reinforce a specific reading skill or concept. Finally, the tutor works with the stu- 
dent to apply the skill or content while the student reads aloud. 

• Data-driven instruction 

Reading Partners uses data to implement and support the model, including the results of 
student assessments that are given three times a year. These assessments are used to create and 
update a student’s Individualized Reading Plan (IRP), which identifies student goals and areas 
on which to focus. 


2 AmeriCorps is a program of the Corporation for National and Community Service that places young 
adults in service positions at nonprofit organizations, schools, public agencies, and faith-based entities. In re- 
turn for their service, AmeriCorps members receive a living stipend, health insurance, and, when they complete 
their program, an education grant. See Corporation for National and Community Service, “Our Programs: 
AmeriCorps,” online publication (2014), at www.nationalservice.gov. 
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• Rigorous and ongoing training 

Reading Partners school-based staff and AmeriCorps members participate in organized 
training sessions before the school year begins as well as ongoing (usually monthly) sessions 
throughout the school year. Topics covered in these training sessions include the Reading Part- 
ners model, a detailed review of the curriculum, how to train volunteer tutors, and how to use 
data and the Reading Partners data system. A short initial training session, as well as periodic 
training on specific topics, is offered to tutors as well. 

• Instructional supervision and support 

Instructional supervision and support are provided on an ongoing basis by and for 
Reading Partners staff and volunteers. Site coordinators supervise volunteers during tutoring 
sessions and provide guidance and suggestions to tutors who need additional support. Program 
managers — more experienced staff members who generally have a background in teaching — 
work with site coordinators to troubleshoot a range of issues, including those related to commu- 
nications with school staff, managing tutors, or identifying best practices to better support the 
progress of a specific student. 


Key Findings 

• The Reading Partners program was implemented with fidelity. 

Despite the myriad difficulties inherent in operating a program whose direct service 
providers are volunteers, Reading Partners was implemented in the schools with a relatively 
high degree of fidelity. 

Students received regular one-on-one tutoring in a dedicated school space. On aver- 
age, students in the study received approximately 1.5 tutoring sessions per week, and spent 28 
weeks in the Reading Partners program. Although this intensity is slightly less than the pro- 
gram model recommends, on average students consistently received three tutoring sessions 
every two weeks. All sites in the study had a designated reading center where tutoring took 
place and where selected materials and resources were made available to program staff, tutors, 
and students. 

Students ’ reading progress was monitored regularly. Ninety-five percent of the pro- 
gram group students who participated in the Reading Partners program for the entire year were 
assessed by Reading Partners staff using their own internal assessments at the three prescribed 
points that the Reading Partners model dictates: when a student first enters the program, mid- 
way through the school year, and at the end of the school year. 
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Staff and volunteers believed that they had adequate training and support to perform 
their jobs successfully. Although the volunteers received limited training before they began 
tutoring (approximately an hour of orientation and tutoring observation), most tutors indicated 
that they felt adequately trained for the role. They also felt well supported by the site coordi- 
nators, who provided monitoring and assistance during tutoring sessions as well as additional 
feedback on how to address specific issues with students outside of the tutoring session. The 
full-time Reading Partners staff and AmeriCorps members (including program managers and 
site coordinators) consistently indicated that they had access to ongoing support from their 
supervisors. 

The biggest challenge that Reading Partners faced in implementing the program was 
ensuring tutor attendance and retention. Reading Partners requests that tutors make a one- 
semester commitment, and site coordinators at the study sites reported that many volunteers, 
particularly high school and college students, did not stay beyond that period of time. As a 
result, new tutors had to be brought on throughout the year. Furthermore, volunteers varied in 
their consistency and commitment. Site coordinators reported that tutors sometimes failed to 
arrive at their scheduled time and at times did not notify the site coordinator beforehand. 
However, Reading Partners put structures in place to address these problems, including the 
use of substitute tutors, make-up days, and tutoring sessions conducted by site coordinators. 
These tactics meant that a student did not necessarily miss tutoring sessions as a result of tutor 
inconsistency. 

• Reading Partners had a positive and statistically significant impact on 
three different measures of student reading proficiency. 

The study quantified the impact of Reading Partners through three different assess- 
ments: the Stanford Achievement Test (SAT- 10) for reading comprehension; the AIMSweb 
oral reading fluency measure; and the Test of Word Reading Efficiency, 2nd Edition (TOWRE- 
2), for sight word efficiency. The effect size of each impact, which reflects the magnitude of the 
difference between the program and control groups, was 0.10 standard deviations for reading 
comprehension, 0.09 for reading fluency, and 0. 1 1 for sight word efficiency. These statistically 
significant impacts are equivalent to approximately one and a half to two months of additional 
progress in reading among the program group members relative to students in the control group, 
who did not participate in the Reading Partners program. 3 

These results (between 0.09 and 0.11 standard deviations) are comparable in magni- 
tude with results of other reading interventions that have been rigorously evaluated on a large 


Statistical significance indicates that the impact is likely a result of the intervention rather than a chance 
occurrence. 
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scale in grades 2 through 5. Although some interventions have produced large impacts for kin- 
dergarteners and first-graders, for students in grades 2 through 5 most rigorous evaluations of 
reading programs generally have found impacts between 0.10 and 0.25 standard deviations, and 
among rigorously evaluated tutoring programs in particular, impacts have been between 0.10 
and 0.15 standard deviations. 4 

Since students in the control group were also receiving supplemental reading services, 
the impact of Reading Partners should be interpreted as the impact of the program relative to 
other supplemental service receipt, not the impact of Reading Partners compared with no inter- 
vention. Approximately two-thirds of students in the control group received at least one sup- 
plemental reading service and were more likely to receive small-group intervention support than 
were their counterparts in the program group. Because the control group students were also 
receiving supplemental reading instruction, the program group members received, in total, only 
about an hour more of reading instruction each week than the control group received. 

Program impacts are robust across a range of student characteristic subgroups as well 
as across subgroups of students with varying levels of reading comprehension skills at baseline. 
Positive and statistically significant impacts were found on at least one measure of reading pro- 
ficiency for each of the following student groups: male students, female students, English lan- 
guage learners (that is, students whose first language is not English), students who are fluent in 
English, students in lower grades (grades 2 and 3), students in upper grades (grades 4 and 5), 
and students with baseline reading comprehension scores in the lowest three quartiles of the 
study sample. 

Reading Partners had a positive and statistically significant impact on the percentage 
of students who moved out of the lowest national quartile in terms of reading comprehension. 
At the end of the year, 1 9 percent of the program group students who had scored in the bottom 
quartile nationally at baseline had moved out of the lowest quartile, as opposed to only 12 per- 
cent of the control group students. 

There were no consistent patterns between the impacts and various school-level 
measures of program context or fidelity. The study team explored the relationship between im- 
pacts and several aspects of program context and fidelity, including tutor consistency, fidelity of 


4 Yung Soo Lee, Nancy Morrow-Howell, Melissa Jonson-Reid, and Stacey McCrary, “The Effect of the 
Experience Corps® Program on Student Reading Outcomes,” Education and Urban Society 44, 1 (2010): 97- 
118; Carrie E. Markovitz, Marc W. Elemandez, Eric C. Hedberg, and Benjamin Silberglitt, Impact Evaluation 
of the Minnesota Reading Corps K-3 Program (Chicago: NORC at the University of Chicago, 2014); Henry 
May, Abigail Gray, Jessica N. Gillespie, Philip Sirinides, Cecile Sam, Heather Goldsworthy, Michael Armijo, 
and Namrata Tognatta, Evaluation of the i3 Scale-up of Reading Recovery: Year One Report, 2011-12 (Phila- 
delphia: Consortium for Policy Research in Education, 2013). 
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implementation, years of operation, and dosage (the frequency and intensity of service deliv- 
ery). Across these analyses, there is no indication of particular aspects of implementation or 
context that made Reading Partners more effective. 

• Reading Partners is a low-cost option for underresourced schools. 

The total resource value, or cost, of Reading Partners is approximately $3,610 per pro- 
gram group student. Other effective early literacy interventions that have been evaluated at scale 
are at least as costly as Reading Partners. 5 However, unlike many other resource-rich programs, 
a majority of Reading Partners’ costs ($1,910 out of $3,610) are in-kind contributions, primarily 
from community volunteers. As a result, Reading Partners schools bear only a small portion of 
the total costs of the program. On average, schools contribute only 20 percent of the total 
resources required to implement the program ($710 per program group student), and over half 
of these costs are in-kind contributions of space and staff time. 6 

The volunteer time and transportation represent the largest portion of the total re- 
sources needed to implement the program. Almost half (42 percent) of the resources required 
for Reading Partners can be attributed to volunteers. Volunteers contributed, on average, the 
equivalent of $1,520 per program group student, which included both their time and transporta- 
tion costs. Because the tutors are not compensated for their time or transportation, they subsi- 
dize a large portion of the costs of the program. 

While Reading Partners is often more resource-intensive than the other supplemental 
services that are available to students in the study schools, many of those resources are provid- 
ed in-kind and thus schools are required to contribute a much smaller portion of those costs. In 
addition to Reading Partners, the six school sites that were included in the cost study offered 
other supplemental services that provided reading instruction beyond what students received 
during regular classroom teaching. Those other services were provided to both program and 
control group students, although control group students received more of them than did the pro- 
gram group. The cost per student for the average of the other supplemental reading services of- 


5 Fiona M. Hollands, Yilin Pan, Robert Shand, Henan Cheng, Henry M. Levin, Clive R. Belfield, Michael 
Kieffer, A. Brooks Bowden, and Barbara Hanisch-Cerda, Improving Early Literacy: Cost-Effectiveness Anal- 
ysis of Effective Reading Programs (New York: Center for Benefit-Cost Studies of Education, Teachers Col- 
lege, Columbia University, 2013); Jessica Simon, “A Cost-Effectiveness Analysis of Early Literacy Interven- 
tions," unpublished paper (New York: Columbia University, 2011). 

6 To calculate the cost per program group student, first the total cost (or the total resource value) of the 
Reading Partners program at each site was divided by the total number of Reading Partners tutoring sessions 
provided to all students in the program during the year (not just the sessions provided to students participating 
in the study), including students in kindergarten and first grade, to obtain the cost per session. Then, to deter- 
mine the average cost of serving each program group student, the average cost per session was multiplied by 
the average number of sessions that the students participating in the study received. 
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fered at the six sites was $1,780 and ranged from $1,050 to $4,890 per student. The range in the 
total resources provided by the supplemental services across the six sites depended on both the 
number and type of supplemental services that were offered. Some services, like computer- 
based programs, were much less resource-intensive. When the cost of Reading Partners is com- 
pared with the cost of other supplemental services that schools offered to struggling readers, the 
cost to the school for Reading Partners was $710 per program group student, as noted above, 
while the average cost of the other supplemental reading services borne by the school or school 
district was $1,700. 


Conclusions 

Overall, the evaluation finds that the Reading Partners model is effective. The program pro- 
duced measurable impacts on reading skills among students who had a fairly broad range of 
reading abilities when the study began and among students from a wide range of grades 
(grades 2 through 5). Reading Partners produced these impacts despite the lack of experience 
among tutors, the somewhat limited training the tutors received, and the relatively high degree 
of tutor turnover. Furthermore, the findings illustrate the high value of the Reading Partners 
program from the perspective of the schools. The program uses many resources, but the vol- 
unteers account for a large part of those resources. As a result, the schools bear only 20 per- 
cent of the costs. 

Thus, the study provides evidence that if the right design and administrative structures 
are put into place, volunteer tutoring programs can be effective when implemented at scale, and 
volunteer tutoring programs may be a cost-effective option for underresourced schools. 

Reading Partners manages to be effective even in the absence of oft-cited key compo- 
nents to successful tutoring, including, in particular, extensive tutor training and tutor consisten- 
cy. 7 Further research is required to understand whether improving these components of the 
model would affect the magnitude of the impacts and whether the impact of Reading Partners is 
sustained for more than one year. 


7 Batya Elbaum, Sharon Vaughn, Marie Tejero Hughes, and Sally Watson Moody, “How Effective Are 
One-to-One Tutoring Programs in Reading for Elementary Students at Risk for Reading Failure? A Meta- 
Analysis of the Intervention Research," Journal of Educational Psychology 92, 4 (2000): 605-619; Gary W. 
Ritter, Joshua H. Barnett, George S. Denny, and Ginger R. Albin, “The Effectiveness of Volunteer Tutoring 
Programs for Elementary and Middle School Students: A Meta-Analysis,” Review of Educational Research 79, 
1 (2009): 3-38; Robert E. Slavin, Cynthia Lake, Susan Davis, and Nancy A. Madden, “Effective Programs for 
Struggling Readers: A Best-Evidence Synthesis,” Educational Research Review 6, 1 (2011): 1-26; Barbara A. 
Wasik, “Using Volunteers as Reading Tutors: Guidelines for Successful Practices,” The Reading Teacher 51, 7 
(1998): 562-570. 
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Chapter 1 

Introduction and Evaluation Overview 


Reading skills are the key building blocks of a child’s formal education. Yet, the national statis- 
tics on literacy attainment are profoundly distressing: 1 Two out of three American fourth- 
graders are reading below grade level and almost one-third of children nationwide lack even 
basic reading skills. For children in low-income families, the numbers are even more troubling, 
with 80 percent reading below grade level. Despite several decades of education reform efforts, 
only incremental progress has been made in addressing this reading crisis. From 1998 to 2013, 
the number of low-income fourth-graders reading at a proficient level increased by only 7 per- 
centage points. 

The Reading Partners program that is described in this report uses community volun- 
teers to provide one-on-one tutoring to struggling readers in underresourced elementary schools. 
Established in 1999 in East Menlo Park, California, Reading Partners is a not-for-profit corpora- 
tion whose mission is to help children become lifelong readers by empowering communities to 
provide individualized instruction with measurable results. The Reading Partners model is 
based on the premise that too many children in low-income communities are not reading profi- 
ciently and that many teachers, schools, and parents in those communities lack the resources 
and infrastructure to address the problem. For these reasons, the program typically serves stu- 
dents in federally designated low-income schools and focuses on using community volunteers 
to provide the needed support for struggling readers. 

At each school, Reading Partners transforms a dedicated space into a “reading cen- 
ter,” places a full-time staff member on site to manage day-to-day operations, and recruits a 
corps of 40 to 100 community volunteers to work one-on-one with struggling readers in kin- 
dergarten through grade 5 (although this evaluation included only students in grades 2 
through 5). Operating independently within the school building, Reading Partners has grown 
to serve more than 7,000 students in over 130 schools throughout California, Colorado, Mary- 
land, New York, Oklahoma, South Carolina, Texas, Washington, and Washington, DC, in the 
years since its inception. 2 

In March 2011, Reading Partners received a three-year True North Fund investment of 
up to $3.5 million in grants from the Edna McConnell Clark Foundation and the Social Innova- 
tion Fund, matched by $3.5 million from True North Fund co-investors, to further expand its 


'U.S. Department of Education, National Center for Education Statistics (2013). 
^Reading Partners (2013). 
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early-intervention literacy program to elementary schools throughout the country and evaluate 
its effectiveness. Box 1 . 1 contains more infonnation about the EMCF Social Innovation Fund. 

A policy brief released by MDRC in June of 2014 reported the initial findings from 
the evaluation, which was conducted during the 2012-2013 school year in 19 schools in three 
states and involved more than 1,100 students. It found positive impacts of the program on 
three different measures of reading proficiency. 3 The findings suggest that Reading Partners 
can be a valuable source of support for elementary grade students who are struggling with 
reading in underresourced schools and deserves wider use in school reform efforts to help im- 
prove reading proficiency. 

This report builds on those initial findings by describing the Reading Partners program 
and its implementation in greater detail, exploring whether or not the program is more or less 
effective for particular subgroups of students, and assessing some of the potential explanations 
for the program’s success. In addition, this report includes an analysis of the cost of implement- 
ing the Reading Partners program in 6 of the 19 sites; it additionally compares the cost of Read- 
ing Partners with the cost of the other supplemental reading services that these six schools 
offered to their students. 

The report concludes that the program is effective for a wide range of student sub- 
groups, including students who are officially designated by the school to be English language 
learners, students starting the program with the weakest skills, and both boys and girls. Fur- 
thermore, although Reading Partners is a resource-intensive program involving the time of 
many individuals, the volunteers themselves account for much of the cost of the program. 
Thus, the cost of implementing the program for schools and districts is quite low. In fact, the 
required school contribution for Reading Partners is substantially lower than it is for the other 
supplemental reading services that are available at the schools, precisely because the volun- 
teers account for a large proportion of the program’s costs. Although further research is war- 
ranted, the findings suggest that the regular one-on-one attention that the students in the 
program group received may have been one of the key contributors to the positive impacts of 
the program. 


The Policy Context 

Reading Partners is being implemented and evaluated in a national context in which increasing 
emphasis is being placed on interventions that are designed to improve reading instruction for 


3 Jacob, Smith, Willard, and Rifkin (2014). 
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Box 1.1 

The EMCF Social Innovation Fund 


The Edna McConnell Clark Foundation (EMCF), in collaboration with MDRC and The 
Bridgespan Group, is leading a Social Innovation Fund (S1F) project that aims to expand the 
pool of organizations with proven programs that can help low-income young people make 
the transition to productive adulthood. The S1F, an initiative enacted under the Edward M. 
Kennedy Serve America Act, targets millions of dollars in public-private funds to expand 
effective solutions across three issue areas: economic opportunity, healthy futures, and youth 
development and school support. This work seeks to create a catalog of proven approaches 
that can be replicated in communities across the country. The S1F generates a 3:1 private- 
public match, sets a high standard for evidence, empowers communities to identify and drive 
solutions to address social problems, and creates an incentive for grant-making organizations 
to target funding more effectively to promising programs. Administered by the federal Cor- 
poration for National and Community Service (CNCS), the SIF is part of the government’s 
broader agenda to redefine how evidence, innovation, service, and public-private coopera- 
tion can be used to tackle urgent social challenges. 

The EMCF SIF is particularly focused on young people who are at greatest risk of failing or 
dropping out of school or of not finding work, who are involved or likely to become involved 
in the foster care or juvenile justice system, or who are engaging in risky behavior such as 
criminal activity or sexual activity that could lead to teenage pregnancy. 

EMCF, with its partners MDRC and Bridgespan, selected an initial cohort of nine programs 
and a second cohort of three programs to receive SIF grants: BELL (Building Educated Lead- 
ers for Life), Center for Employment Opportunities, Children’s Aid Society— Carrera Adoles- 
cent Pregnancy Prevention Program, Children’s Flome Society of North Carolina, Communi- 
ties in Schools, Gateway to College Network, PACE Center for Girls, Reading Partners, The 
SEED Foundation, WINGS for Kids, Youth Guidance, and Children’s Institute, Inc. These or- 
ganizations were selected through a competitive process based on prior evidence of impacts on 
economically disadvantaged young people, a track record of serving young people in commu- 
nities of need, strong leadership and a potential for growth, and the financial and operational 
capabilities necessary to expand to a large scale. 

The EMCF Social Innovation Fund initiative, called the True North Fund, includes support 
from CNCS and 15 private co-investors: The Edna McConnell Clark Foundation, The An nie 
E. Casey Foundation, The Duke Endowment, The William and Flora Hewlett Foundation, The 
JPB Foundation, George Kaiser Family Foundation, The Kresge Foundation, Open Society 
Foundations, The Penzance Foundation, The Samberg Family Foundation, The Charles and 
Lynn Schusterman Family Foundation, The Starr Foundation, Tipping Point Community, The 
Wallace Foundation, and the Weingart Foundation. 
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early elementary school students. Several national initiatives have been instituted in recent years 
(such as the Reading Excellence Act of 1997 and Reading First) in an attempt to provide 
schools with better resources to support their struggling elementary students. In addition, a 
number of states have either enacted or are considering enacting legislation regarding third 
grade reading proficiency, with 14 states and Washington, DC, having adopted laws that require 
students to be retained if they are not reading at grade level by the end of third grade. 4 This poli- 
cy environment has implications for interpreting the results of the evaluation because it affects 
the degree to which other supplemental services are available to students in the study. 

The No Child Left Behind Act of 2002 requires that schools provide low-income fami- 
lies with extra academic assistance or supplemental education services if their children are 
attending a Title I school that is in Program Improvement. 5 These supplemental education ser- 
vices come at no cost to families and can include tutoring or remedial help in subjects like read- 
ing, language arts, and math, as long as they are aligned with state content standards and 
grounded in high-quality research that provides evidence of their effectiveness. 6 Students can 
receive this extra help, which is often provided by external programs and vendors, before or 
after school, on weekends, or in the summer. 7 

Many of these supplemental education services use Response to Intervention (Rtl) 
models, which provide a means of identifying and supporting struggling students. Under Rtl, an 
approach for which funding was authorized under the Individuals with Disabilities Education 
Improvement Act of 2004, all students are monitored and assessed for placement into a multi- 
tiered system of intervention. Students who are identified as struggling learners are supported 
by an array of interventions at increasing levels of intensity. Increasing the intensity of an inter- 
vention can be accomplished in a number of ways, including conducting an intervention more 
frequently or adding to its duration, relying on more specialized instructors, or pulling students 
into smaller groups (“pull-out” programs). The progress of each at-risk student is monitored 
closely using research-based assessments to make decisions about intervention duration and 
intensity. Tier I, usually classroom instruction for all students, is considered the first level of 
intervention, with Tier II and Tier III being progressively more intense. 8 For example, daily 
one-on-one work with a specially trained reading interventionist is considered a Tier III inter- 


pose (2012). 

5 Title I schools are those that receive federal funds based on the number of low-income children in attend- 
ance. Funds are used to improve academic achievement in reading and math in a variety of ways — offering 
teacher professional development, hiring additional teachers, or modifying curricula. A school in Program 
Improvement has been designated by the state to be in need of improvement for more than one year (U.S. 
Department of Education, 2012). 

6 Heinrich, Meyer, and Whitten (2010). 

U.S. Department of Education (2012). 

8 Fuchs and Fuchs (2006). 
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vention. Reading Partners is typically considered a Tier II intervention (for students needing 
some additional support), although some schools report using Reading Partners as a Tier III 
intervention. 9 Reading Partners is usually one among multiple Tier II interventions that are 
available for struggling readers at any given school. 


Evidence-Based Approaches to Improving Literacy 
in Elementary School 

A variety of interventions exist to help students who are learning to read in elementary school. 
Among them, two models — whole-school reform and one-on-one tutoring by trained teachers 
— have received widespread attention, in part because they have been rigorously evaluated at 
scale (meaning that they have been tested on hundreds of students in multiple schools) and 
found to be effective in improving reading skills. A third model, tutoring by volunteers or 
paraprofessionals, used by Reading Partners, has received less scrutiny. 

Whole-School Reform 

Whole-school reform involves schoolwide changes to the curriculum, assessments, and 
interventions for struggling students in an ahempt to change a school’s instructional culture and 
practices. The approach is exemplified by the Success for All program, in which teachers im- 
plement a highly structured reading curriculum, cooperative learning techniques, whole-school 
improvement practices, frequent assessments, and daily one-on-one tutoring for struggling stu- 
dents. The Success for All program has been widely evaluated and has been involved in two 
large-scale randomized controlled trials, all of which have found positive impacts of the pro- 
gram on students’ reading skills. 10 

One-on-One Tutoring Delivered by Trained Teachers 

One-on-one tutoring delivered by certified teachers has consistently demonstrated large 
impacts on the reading proficiency of struggling readers. 11 Among the best known and most 
widely researched of these programs is Reading Recovery. Reading Recovery is a short-term 
early intervention designed to help the lowest-achieving 15 percent to 20 percent of readers in 
first grade by providing daily one-on-one tutoring. Reading Recovery teachers, who receive 

9 A 2011 Reading Partners survey asked principals to indicate how Reading Partners was used in their 
schools. Of the 32 principals who answered the question about whether they used Reading Partners as an Rtl 
intervention in their schools, 81 percent indicated that they did. Of the 26 who answered a question about what 
tier they used Reading Partners tutoring services for, 50 percent said they used Reading Partners as a Tier II 
intervention, and another 35 percent said they used Reading Partners as a Tier II/III intervention. 

10 Borman et al. (2007); Quint et al. (2013); Quint et al. (2014). 

1 1 Slavin. Lake, Davis, and Madden (2011); May et al. (2013). 
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extensive and ongoing training, provide participating students with daily 30-minute lessons tai- 
lored to individual student needs over a period of 12 to 20 weeks. 

Tutoring by Volunteers or Paraprofessionals 

While the whole-school reform approach and one-on-one tutoring by trained teachers 
each has a solid research base demonstrating its effectiveness, they are both time- and resource- 
intensive ways to ameliorate the problem of low literacy, and as such may not always be viable 
options for already underresourced schools. 

A third, less validated approach involves using volunteers or paraprofessionals, as 
opposed to trained teachers, to provide tutoring to struggling readers. Tutoring by volunteers 
or paraprofessionals has the potential to be a less costly method for addressing the problem of 
low literacy, but to date, little rigorous evidence exists regarding the efficacy of this approach. 
Although available evidence on the effectiveness of one-on-one tutoring provided by volun- 
teers is encouraging, there is currently only limited evidence that such programs can be im- 
plemented effectively on a large scale. Among seven studies of volunteer tutoring programs 
that were reviewed in a meta-analysis, only two were evaluated experimentally with interven- 
tion groups larger than 100 students. Only one of those two, the Experience Corps program, 
was found to have positive impacts. 12 Since then, Minnesota Reading Corps has also demon- 
strated effectiveness at scale. 13 Furthermore, prior research on the implementation and effec- 
tiveness of programs that rely largely on volunteers generally suggests that bringing them to 
scale might be quite difficult. 14 

The evaluation described in this report is designed to bring more information to bear on 
the potential effectiveness of such volunteer tutoring programs. This report reflects a third rig- 
orous evaluation of a volunteer tutoring program delivered on a large scale (the other two being 
the evaluations of Experience Corps and Minnesota Reading Corps, described above), and the 
Reading Partners program is unique among these evaluations in that it relies on community vol- 
unteers who devote only a few hours each week to tutoring (as opposed to Minnesota Reading 
Corps, which primarily relies on AmeriCorps members who commit to tutoring full time or half 
time, and Experience Corps, an AmeriCorps affiliate in which senior-age volunteers commit up 
to 15 hours a week to tutoring). 15 It is intended to provide rigorous evidence about the efficacy 
of the Reading Partners program implemented at scale. Furthermore, it assesses the cost of 


12 Slavin, Lake, Davis, and Madden (2011). 

°Markovitz, Hernandez, Hedberg, and Bilberglitt (2014). 

14 Grossman and Furano (1999); Hager and Bnidney (2004). 

15 AARP Experience Corps (2014); Markovitz, Hernandez, Hedberg, and Bilberglitt (2014). 
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implementing a volunteer tutoring program and explores the cost of implementing the program 
from the perspective of a school. 


Overview of the Evaluation 

To answer questions about the effectiveness of the Reading Partners program and the potential 
for volunteer tutoring more generally to help improve the reading proficiency of struggling 
readers, this evaluation includes an implementation study, an impact study, and a cost study. 
The implementation and impact studies included 19 schools with Reading Partners sites in three 
different states, and the cost study included a subsample of six of those sites. Together, these 
three facets of the evaluation are designed to address the following broad research questions: 

1. In what context was the Reading Partners program implemented, and was it im- 
plemented as intended or with fidelity (that is, with adherence to the program 
model as laid out by the program developers)? How much variability in fidelity of 
implementation was observed across the sites? What factors contributed to any 
observed variability? 

2. On average, did the Reading Partners program have a positive impact on students’ 
reading proficiency across three key components of early reading ability: sight 
word efficiency, reading fluency, and comprehension? 

3. What resources are needed to implement the Reading Partners program as de- 
scribed in this report and what proportion of the costs of implementing the pro- 
gram are borne by the school? 

Implementation Study 

The implementation study included site visits to all the schools participating in the 
evaluation in the winter of the study year, interviews with key program and school staff and 
volunteers, and the collection of programmatic data from the sites and from Reading Partners’ 
own management information system. It was designed to enable the study team to describe the 
program and the context in which it was implemented in detail and to assess whether the pro- 
gram was delivered as intended. In addition, it was designed to explore whether any specific 
factors helped facilitate the implementation of the program or posed barriers to effective imple- 
mentation, and how sites addressed those challenges. Implementation studies such as this one 
are essential for understanding the results of impact evaluations and contribute valuable infor- 
mation on the generalizability of the findings, the feasibility of implementing the program in 
other locations, and elements that might facilitate or strengthen future implementation. 
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Impact Study 

The impact study was designed to test the effect of the Reading Partners program on 
student reading proficiency — that is, to determine whether the tutoring provided by the Read- 
ing Partners’ volunteers increased the reading skills of the students who participated. The study 
used a student-level randomized controlled trial design, in which students were randomly as- 
signed within each school to either a program group that would participate in Reading Partners 
during the 2012-2013 school year or an “as is” control condition without Reading Partners. 
Random assignment helps ensure that there are no systematic differences between the two 
groups of students at baseline, so that any positive effects can be causally attributed to the pro- 
gram with a high degree of confidence. Any differences between the outcomes of the program 
and control groups at the end of the study period are considered impacts of the program. A total 
of 1,265 students in grades 2 through 5 across the 19 schools were randomly assigned (646 to 
the program group and 619 to the control group). Although Reading Partners typically serves 
students in kindergarten through grade 5, the funding priorities of the granting institution, 
EMCF, typically focus on older children and adolescents. For that reason, the decision was 
made to include only second- through fifth-graders in this evaluation. 

At the start of the school year, three assessments were given to all students in the study 
sample. The assessments measured reading comprehension, fluency, and sight word efficiency, 
all key components of early reading. 16 These same three assessments were administered to stu- 
dents again in the spring, as close to the end of the school year as possible. 

Cost Study 

The cost study describes the cost of implementing the Reading Partners program during 
the evaluation, based on a comprehensive list of the resources (or “ingredients”) used in imple- 
mentation, and analyzes who bears the burden of financing those costs. This method works by 
calculating an intervention’s total cost by summing the cost of all the resources needed to im- 
plement the program, regardless of who financed them. 17 The analysis uses the same approach 
to examine the costs of the other supplemental reading services at a sample of study schools. 
This approach provides a rich description of the costs of Reading Partners, as well as the costs 
of the other supplemental reading services in a sample of the evaluation schools, and allows for 
a detailed analysis of who absorbs those costs and how those costs vary across sites. 


l6 Reading comprehension was assessed using the Stanford Achievement Test 10th Edition (SAT-10) read- 
ing comprehension subtest; fluency, the ability to read with speed and accuracy, was assessed using the 
AIMSweb one-minute oral reading fluency subtest; and sight word efficiency, the ability to quickly identify 
commonly used words without going through the process of decoding, was assessed using the Test of Word 
Reading Efficiency, 2nd Edition (TOWRE-2) sight word reading subtest. 

17 Levin (1975); Levin and McEwan (2001). 



For budgetary reasons, only a subsample of schools could be included in the cost study. 
The team selected sites with strong implementation, that were geographically representative of 
the other sites in the study, and where reliable data could be collected on the resources used in 
Reading Partners and other supplemental reading services offered during the 2012-2013 school 
year. In order to execute this component of the study, MDRC partnered with the Center for 
Benefit-Cost Studies of Education at Teachers College, Columbia University. Cost studies like 
these are essential for helping researchers, practitioners, and programs themselves understand 
what resources are required for implementing a program. 

Evaluation Sample of Schools 

The sample for this study consists of 19 schools with Reading Partners reading centers 
that were recruited for the evaluation after Reading Partners senior staff identified them as poten- 
tial participants. Only schools in which Reading Partners had been operating for at least one year 
before the study began were eligible, although the staff in these schools included both new and 
returning staff members. Schools were also eligible to participate only if Reading Partners antici- 
pated that they would not be able to provide services to all the students who needed help with 
reading. This stipulation ensured that each program site was able to serve the same number of 
eligible students as it typically would, and did not deny services as a result of random assign- 
ment. A total of 31 schools were contacted during recruitment, and 19 of them joined the study. 

The 19 participating schools were spread across 12 school districts and 6 Reading Part- 
ners programmatic regions (New York City; Washington, DC; and, in California, East Bay, 
South Bay, Sacramento, and Los Angeles). 18 There were 16 schools in California, 2 in New 
York, and 1 in Washington, DC. All of the schools were established Reading Partners sites; 8 of 
the 19 schools were in their second year of operating a Reading Partners center, while the rest 
had been operating a Reading Partners center for at least three years. The schools in the Reading 
Partners study sample had high numbers of students receiving free or reduced-price lunch (a 
proxy for coming from a low-income family), minority students, and English language learners. 
The majority of the participating schools were schoolwide Title I schools. Eight of the study 
schools were in varying stages of federal School Improvement status and two were in the final 
year of a three-year School Improvement Grant (grants to state education agencies to support 
school improvement goals in the nation’s lowest-performing schools). 

Table 1.1 shows the characteristics of the study sample compared with other Reading 
Partners schools that were not included in the sample, with all federally designated Title I 


18 While 19 schools participated in the study, there were only 17 distinct Reading Partners centers. In the 
study, two sets of two colocated schools shared a reading center. 
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Reading Partners Evaluation 
Table 1.1 

Characteristics of Reading Partners Study Schools and Other School Samples (2011-2012) 


Study Other Reading Partners Schools 11 U.S. Title I Schools'^ Average U.S. Schools" 


Characteristic 

Schools 

Mean 

Difference 

P-Value 

Mean 

Difference 


P-Value 

Mean 

Difference 


P-Value 

Eligible for Title I program (%) 

88.89 

96.20 

-7.31 

0.209 

100.00 

- 11.11 

*** 

<0.001 

75.39 

13.49 


0.184 

Students eligible for 
ffee/reduced-price lunch d (%) 

81.70 

84.52 

-2.82 

0.411 

69.06 

12.65 

*** 

0.008 

53.47 

28.23 

*** 

<0.001 

Race/ethnicity (%) 













Black 

20.74 

28.44 

-7.71 

0.304 

20.74 

0.00 


1.000 

15.92 

4.82 


0.404 

Hispanic 

61.75 

52.95 

8.80 

0.204 

29.83 

31.92 

*** 

<0.001 

23.34 

38.40 

*** 

<0.001 

Asian 

9.60 

8.30 

1.29 

0.639 

3.43 

6.16 

*** 

0.002 

4.67 

4.92 

** 

0.025 

White 

5.70 

7.28 

-1.58 

0.496 

41.28 

-35.58 

*** 

<0.001 

51.60 

-45.90 

*** 

<0.001 

Other 

2.22 

3.03 

-0.81 

0.336 

4.73 

-2.51 


0.263 

4.47 

-2.25 


0.234 

Male (%) 

51.89 

51.82 

0.07 

0.970 

53.54 

-1.64 


0.462 

53.00 

-1.1 


0.564 

Average number of students 

494 

451 

43 

0.318 

468 

26 


0.622 

472 

22 


0.677 

Grade 2 

79 

69 

10 

0.169 

68 

10 


0.217 

70 

8 


0.331 

Grade 3 

76 

65 

11 

0.142 

68 

8 


0.384 

70 

5 


0.554 

Grade 4 

74 

65 

9 

0.227 

67 

7 


0.396 

70 

5 


0.591 

Grade 5 

76 

64 

12 

0.131 

67 

9 


0.290 

69 

6 


0.488 

School setting 0 (%) 













Urban 

73.68 

91.67 

-17.98 ** 

0.027 

39.77 

33.91 

*** 

0.003 

31.77 

41.91 

*** 

<0.001 

Suburban 

26.32 

8.33 

17.98 ** 

0.027 

21.28 

5.03 


0.592 

29.50 

-3.19 


0.761 

Town 

0.00 

0.00 

0.00 

NA 

10.07 

-10.07 


0.145 

8.79 

-8.79 


0.176 

Rural area 

0.00 

0.00 

0.00 

NA 

28.87 

-28.87 

*** 

0.005 

29.94 

-29.94 

*** 

0.004 

Sample size 

19 

84 



24,914 




42,202 





(continued) 


Table 1.1 (continued) 

SOURCE: MDRC calculations from 2011 and 2012 National Center for Education Statistics Common 
Core of Data (CCD). 

NOTES: Sample sizes for individual outcomes may fall short of the reported sample sizes because of 
missing or unusable data. 

A two-tailed t-test is used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

a "Other Reading Partners Schools" include all other nonstudy Reading Partners schools that meet 
the "Average U.S. Schools" criteria below. 

b "U.S. Title I Schools" include all non-Reading Partners schools that meet the "Average U.S. 
Schools" criteria below and were all designated Title I schoolwide schools. 

c " Average U.S. Schools" include non-Reading Partners schools that offer grade 2 through grade 5, 
are defined as "regular" schools by the CCD, and are located within the 50 U.S. states and the District 
of Columbia. 

d The value given for students eligible for free/reduced-price lunch is calculated from the 2011 CCD 
because data are missing in the 2012 CCD. Data for all other variables are from the 2012 CCD. 

e "Urban" is defined as territory inside an urbanized area and inside a principal city having a 
population greater than 100,000. "Suburban" is defined as territory outside a principal city and inside 
an urbanized area with a population of less than 250,000. “Town” is defined as territory inside an 
urban cluster that is (1) less than or equal to 10 miles from an urbanized area (“fringe”), or (2) more 
than 10 miles and less than or equal to 35 miles from an urbanized area (“distant”), or (3) more than 35 
miles from an urbanized area (“remote”). “Rural” is defined as territory that is (1) less than or equal to 
5 miles from an urbanized area and less than or equal to 2.5 miles from an urban cluster (“fringe”), or 
(2) more than 5 miles but less than or equal to 25 miles from an urbanized area and more than 2.5 
miles but less than or equal to 10 miles from an urban cluster (“distant”), or (3) more than 25 miles 
from an urbanized area and more than 10 miles from an urban cluster (“remote”). 


schools in the United States, and with all elementary schools in the United States. 19 The Read- 
ing Partners study schools were somewhat less urban (and more suburban) than Reading Part- 
ners schools as a whole, but there were no other statistically significant differences between the 
two types of schools. (This discussion refers only to statistically significant differences — that 
is, differences that are unlikely to have arisen by chance.) The Reading Partners study schools 
include a higher percentage of Hispanic students than the average Title I school, reflecting the 
large concentration of California schools in the study. The study sample also includes more ur- 
ban schools than are represented among all schoolwide Title I schools, since Reading Partners 
does not attempt to serve small, rural schools, for which the number of available volunteers is 
quite limited. Consistent with the Reading Partners model, the Reading Partners schools in the 
study sample are poorer, more urban, and have higher percentages of minority students than the 


l 9 Title I schools are eligible for additional funding because of their high proportion of students from low- 
income families. 
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population of U.S. schools as a whole. Thus, the sample of schools included in the evaluation 
appears to be broadly representative of Reading Partners sites across the country. 


The remainder of this report is organized as follows: Chapter 2 provides a detailed 
overview of the Reading Partners program. Chapters 3, 4, and 5 present the findings from the 
implementation, impact, and cost components of the study, respectively, and Chapter 6 offers 
some interpretations and conclusions. 
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Chapter 2 

The Reading Partners Program 


This chapter provides a detailed description of the Reading Partners program model as it was 
implemented during the 2012-2013 school year, including its staffing structure and core com- 
ponents. The information provided in this chapter is a combination of the model as described by 
Reading Partners and as understood from data collected over the course of this study. Reading 
Partners has continued to innovate and refine its program model since that time and thus this 
description may not fully reflect current operating practices. Figure 2.1 provides an overview of 
the program in its ideal implementation. 

Reading Partners is a “pull-out” program that operates both during the school day and 
after school. 1 The highly structured, modular curriculum is delivered by volunteer tutors on a 
one-on-one basis in 45-minute sessions, twice a week. Each school that participates in the Read- 
ing Partners program has a designated “reading center” where the tutoring takes place. National- 
ly, the average student is enrolled in the program for more than five months, and many students 
participate in the program for a full school year or longer. The curriculum used by Reading 
Partners during the 2012-2013 study year was research-based and aligned with California state 
content standards. From 2012 to 2014, Reading Partners has worked to systematically align the 
curriculum with the Common Core State Standards, which establish a single set of suggested 
national educational standards for students in kindergarten through twelfth grade in English lan- 
guage arts/literacy and mathematics. 2 


Target Population and Selection Criteria 

Reading Partners aims to serve students in kindergarten through grade 5 in underresourced 
schools who are half a year to two and a half years behind grade level in reading. In addition, 
Reading Partners aims to serve students who are conversant in English and do not have any 
special needs (that is, do not have an Individualized Education Program). 3 Many Reading Part- 
ners students are in fact designated by their schools as English language learners (ELLs) — 


'“Pull-out” refers to a type of program that takes students out of the mainstream classroom or after-school 
program for a portion of the day, with the goal of providing specialized instruction. 

‘Since 2010, 46 states have adopted the Common Core (National Council of State Legislatures, 2014). 
'individualized Education Programs are developed for children who are found through assessment to have 
a disability that affects their learning process. The program outlines how teachers will help these students leam 
more effectively considering each of their learning styles and needs. 
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Reading Partners Evaluation 
Figure 2.1 

Reading Partners Program 


Schools refer eligible students to Reading Partners. 

Reading Partners targets students who: 

• Are 6-30 months (2.5 years) below grade level in reading 
proficiency 

• Have at least conversational English skills 

• Do not have an Individualized Education Program 



^ ^ 

Reading Partners administers an assessment to confirm 

target students and place them into the curriculum. 
Tutors are then matched with students. 


Reading Partners staff and 
volunteer tutors receive 
ongoing training. 


Site coordinators and volunteer 
tutors receive instructional 
supervision and support. 



Students participate in twice-weekly Reading 
Partners sessions that include: 

One-on-one tutoring from a volunteer tutor 
Data-driven instruction, monitored by the site 
coordinator 

Structured, individualized curriculum 
Instruction in a dedicated space 



▼ 


Reading Partners administers a midyear student 
assessment to monitor student progress and modify 
the student’s Individualized Reading Plan. 



T 

Student reading proficiency 
measured at end of year to 
determine overall growth. 
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however, ELL students who are enrolled in Reading Partners are generally conversationally 
fluent in English. 

Students are referred to Reading Partners by teachers or principals. 4 At each site, teach- 
ers, principals, and reading specialists identify students who are reading below grade level and 
consider the different reading interventions that are available to students. Often, Reading Part- 
ners is one program on a menu of supplemental services that are available to students in under- 
resourced schools, particularly if the school has been designated under the No Child Left Be- 
hind act to be “in need of program improvement” for more than one year. 5 School staff mem- 
bers often use a combination of information to determine whether or not a student should be 
referred to Reading Partners, including students’ state standardized test scores, students’ per- 
formance on local assessments, academic performance in the classroom, and behavior (based on 
teacher observation). 


Reading Partners’ Organizational Structure, Staff, 
and Volunteer Tutors 

An overview of the Reading Partners staffing structure that supports program delivery at each 
Reading Partners center is provided in Figure 2.2. Each Reading Partners center is managed by 
a site coordinator. Site coordinators oversee the instruction that the volunteer tutors provide and 
the day-to-day operations of the Reading Partners program at school sites, managing 40 to 100 
volunteer tutors over the course of the year. The number of active volunteer tutors at any one 
point in time, however, can vary significantly. In addition, site coordinators provide training and 
support to volunteer tutors; use Reading Partners materials, tools, and activities to ensure that 
students’ learning needs are met; and serve as liaisons with school staff (teachers, reading coor- 
dinators, administrators, and so forth). Site coordinators also tutor students directly, especially 
when a substitute tutor is needed and there are more students than available tutors. 

Site coordinators are almost exclusively AmeriCorps members and most are recent 
college graduates. 6 The site coordinator position is intended to be a two-year post. All site 


4 To accommodate the study, school staff members were asked to identify and refer a pool of eligible stu- 
dents to Reading Partners earlier in the school year than usual. From this pool, all eligible students were ran- 
domly assigned into the program, rather than hand-picked and enrolled into the program on an ongoing, rolling 
basis, as is typical. 

5 U.S. Department of Education (2012). 

6 AmeriCorps is a program of the Corporation for National and Community Service that places young 
adults in service positions at nonprofit organizations, schools, public agencies, and faith-based entities. In re- 
turn for their service, AmeriCorps members receive a living stipend, health insurance, and, when they complete 
their program, an education grant (Corporation for National and Community Service, 2014). 
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Reading Partners Evaluation 


Figure 2.2 

Staffing Support Structure for a Reading Partners Center 



NOTE: Within Reading Partners, regional executive directors supervise multiple program 
managers, and program managers supervise multiple site coordinators. Additionally, 
outreach coordinators are responsible for recruiting volunteers for multiple site 
coordinators. 


16 






coordinators report to and are supported by program managers, who often have classroom 
teaching experience. Program managers are charged with being the primary literacy program 
expert for a portfolio of school sites and manage a team of up to five site coordinators. The pro- 
gram manager’s roles and responsibilities include building and cultivating relationships with 
schools, districts, and other community organizations; conducting monthly training sessions for 
volunteer tutors and site coordinators; ensuring that Individualized Reading Plans (IRPs) for all 
students are created; and ensuring that data are used to maximize students’ reading skills devel- 
opment. Another key responsibility of program managers is to visit the reading centers in their 
portfolio of school sites regularly to monitor the implementation of the programs at each site, 
and to provide support where needed — by, for example, communicating with school staff. 

Outreach coordinators also play a key role in the Reading Partners model. However, 
their primary role does not involve direct interaction with students. They are responsible for re- 
cruiting volunteers year-round in order to meet the need for volunteers at each program site. 
Outreach coordinators, who are also AmeriCorps members, oversee the entire recruitment pro- 
cess and are responsible for initiating contact with every individual who expresses interest in 
volunteering for Reading Partners. As part of the recruitment effort, outreach coordinators are 
also charged with identifying and establishing relationships with partner organizations such as 
universities, religious institutions, and businesses that can provide a steady stream of tutors. His- 
torically, high schools and postsecondary institutions have served as the largest source of tutors. 

Outreach coordinators and program managers report directly to a regional executive 
director, a senior member of the organization. Regional executive directors oversee programs 
within a given region and are charged with supporting tutor recruitment, ensuring school sup- 
port, and engaging in additional fundraising. 

Volunteer tutors are responsible for providing one-on-one tutoring to Reading Partners 
students and implementing the Reading Partners curriculum. Tutors at the study schools often 
learned about Reading Partners through their schools, through community organizations, or 
through online avenues. Reading Partners publicizes tutoring opportunities on a variety of web- 
sites, including Volunteer Match, Volunteer Center, Go Volunteer, Christian Volunteering, Ide- 
alist, and Craigslist. Others learned about Reading Partners through friends or community or- 
ganizations. All tutors are asked to make a one-semester commitment (generally around four 
months), and as part of that commitment, to identify regular days and times that they can come 
in to work with one or more students every week. Additionally, tutors must pass a background 
check that allows them to work with students. 

No prior experience working with children is necessary to become a Reading Partners 
tutor, and as such the tutor pool is diverse. Tutors range in age (14 to 70 years of age and older), 


17 



Reading Partners Evaluation 
Figure 2.3 

Backgrounds of Tutors in Study Schools 


11 . 1 % 


10.3% 


8 . 1 % 


17.6% 


17.2% 



35.6% 


MHigh school students Q College or graduate students 
^Working adults RH] Retired adults 

[||{j]| Other | [Missing 

SOURCE: MDRC calculations from Reading Partners management information 
system data. 

NOTE: Tutors characterized as "Other" include, but are not limited to, caregivers, 
nonworking individuals, and those who describe themselves as "transitioning." 

gender, and race/ethnicity, and are represented by people from many walks of life. As shown 
in Figure 2.3, about 36 percent of tutors included in this study were college or graduate stu- 
dents, 18 percent were working adults, 17 percent were high school students, 8 percent were 
retired adults, and 10 percent were caregivers, unemployed, or making the transition from be- 
ing unemployed to working; a final 1 1 percent did not provide this information. However, as 
shown in Appendix Table C. 1, the tutor pool composition within each site varied widely, with 
some sites being composed almost entirely of college or graduate students (up to 83 percent in 
one site) and other sites having large proportions of working adults (up to 40 percent) or high 
school students (up to 31 percent). Typically, the composition of the tutor pool was influ- 
enced by proximity to, and relationships built with, local colleges or high schools, and access 
to public transportation. 
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A little less than half (22 of the 49) of the tutors who were interviewed for the study had 
some experience working with children in some capacity, whether it was leading a church youth 
group, helping in an after-school program, working at a children’s summer program, or previ- 
ous teaching experience. However, interviews with volunteer tutors also revealed that fewer 
(11) had experience in an organized tutoring program. 


Core Components of Reading Partners 

The Reading Partners model consists of six core components, each described in turn below. The 
core components are (1) regular, one-on-one tutoring; (2) dedicated school space and use of ma- 
terials; (3) a structured and individualized curriculum; (4) data-driven instruction; (5) rigorous 
and ongoing training for staff and volunteers; and (6) instructional supervision and support. 

Regular, One-on-One Tutoring 

A key feature of the Reading Partners model is to provide students with individualized 
reading instruction. As such, volunteer tutors deliver the Reading Partners curriculum to stu- 
dents on a one-on-one basis. The one-on-one support — as opposed to small-group support — 
is the main component of the program that sets Reading Partners apart from other supplemental 
services that are available to struggling readers. The one-on-one sessions are intended to be de- 
livered twice a week for 45 minutes by a volunteer tutor. While not explicitly part of the model, 
the assumption is that a student will also benefit from a supportive relationship with a caring 
adult, suggesting that consistency is an implicit goal. However, as described in more detail be- 
low, tutors often visit schools only once a week for up to a few hours (working with one student 
for 45 minutes or multiple students consecutively), so many students are paired with two differ- 
ent tutors throughout the school year, typically meeting with one tutor on the first day and an- 
other tutor on the second day. 

Dedicated School Space and Use of Materials 

The Reading Partners tutoring is designed to take place on school grounds in a dedicat- 
ed tutoring space — typically a designated classroom — that is transformed into a reading cen- 
ter. Every reading center features a “read-aloud” library where students and tutors can select 
books at different levels of difficulty, a comer of the room with couches or bean bags for read- 
aloud sessions, work stations for tutor-student pairs, a “take-home reading” area where students 
can select books to read at home, and reading-themed materials on the walls. In addition, Read- 
ing Partners provides each site coordinator with a list of items that all Reading Partners centers 
are expected to have — for example, a resource table for volunteers, a “word wall” showing 
common words for students to use as a reference, and a bulletin board to display student work. 
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These components and materials are designed to create an engaging learning environment in 
which multiple tutor-student pairs can be working at the same time. 

Structured and Individualized Curriculum 

The Reading Partners curriculum that was implemented during the study year was 
modularized, with each lesson following a consistent structure. The curriculum consisted of two 
modules: a beginning readers module (consisting of one level) and a comprehension readers 
module (consisting of four levels). 

The beginning readers module (Level 1) contained 50 lessons, which were primarily 
phonics-based and covered various letter-sound combinations and practice with high-frequency 
words. Typically, students were able to get through one beginning readers lesson in one 45- 
minute session. However, some students in this module took up to two sessions to complete a 
lesson. Supplemental lessons were also available for some lessons in case a student needed 
additional practice. 

The comprehension readers module comprised Levels 2, 3, 4, and 5, each of which con- 
sisted of 24 lessons. The lessons in this module focused on topics such as cause and effect, mak- 
ing inferences, fact and opinion, summarizing, and predicting. Typically, students finished a 
comprehension readers lesson in two to three sessions. 

The tutoring session typically began with the tutor picking up the student from the 
classroom. During the walk to the Reading Partners center, the tutor engaged in friendly conver- 
sation with the student, helping to build rapport and a personal connection. The lesson itself fol- 
lowed a consistent structure. Each lesson began with the tutor reading aloud from a text that the 
student chose. During that time, the tutor modeled fluent reading, paying close attention to tone, 
pronunciation, and punctuation. Tutors paused to ask the student comprehension questions or to 
check the student’s knowledge of key vocabulary. After the read-aloud, the tutor turned to the 
Reading Partners curriculum materials, which introduced a new skill or concept or reinforced a 
skill or concept that had been taught previously. The tutor worked with the student to complete 
the task in the curriculum materials. Finally, the student applied the new skill or concept while 
reading aloud from a text that was provided to support the lesson, and with support from the 
tutor. A description of a typical Reading Partners tutoring session can be found in Box 2. 1 . 

Data-Driven Instruction 

Reading Partners uses data to implement and support the model. For example, in 
determining whether or not a student should be referred to Reading Partners, a combination 
of data are considered: student performance on state standardized tests, performance on 
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Box 2.1 


A Reading Partners Tutoring Session 


1. Meeting the student (5 minutes). Tutors pick Reading Partners students up from their 
classrooms or after-school programs and walk with them back to the Reading Partners 
center. During that time, tutors talk with the students about how they’re doing and engage 
in caring and friendly conversation. This approach lays the groundwork for a healthy tu- 
tor-student relationship and helps to build rapport between the tutor and student. 

2. Tutor read-aloud (10 to 15 minutes). Once the tutor-student pair reaches the Reading 
Partners center, the lesson begins with a student choosing a book at the appropriate level 
from the Reading Partners library and finding a quiet space to sit with the tutor. The tutor 
then reads the book aloud to the student, modeling how to read expressively, pay attention 
to punctuation, and pronounce words correctly. During the read-aloud, the tutor pauses to 
ask the student open-ended questions and to discuss the book’s content and vocabulary. 

3. Introduction of a new skill or concept (10 to 15 minutes). The tutor introduces a new 
skill or concept and helps the student complete an introductory task. For example, a 
beginning student may leam about words with an “oa” sound, while a more advanced 
comprehension student may leam about the concept of cause and effect. 

4. Student read-aloud and application of a new skill or concept (15 to 20 minutes). While 
the student reads a literary or informational text aloud, the tutor provides support and en- 
courages the student to apply the new skill or concept by asking targeted questions. 


school-administered assessments (for example, Developmental Reading Assessment, Basic 
Phonics Skills Test), and information provided by teachers on the Reading Partners referral 
form (such as observational assessments of a student’s level of comprehension or ability to 
use a knowledge of letters and sounds to read written words correctly — often referred to as 
“phonics”). 

After the teacher or principal refers the student to Reading Partners, the site coordinator 
or program manager administers the Rigby PM Benchmark assessment, an established tool that 
measures oral fluency, accuracy, and comprehension. 7 Site coordinators and program managers 
receive training and a protocol for administration, and set aside practice time before conducting 
assessments one-on-one with students. Using a dedicated set of reading materials at various lev- 
els of difficulty, the trained administrator asks the student to read the book aloud. The adminis- 
trator measures accuracy by recording any errors or corrections that the student makes during 


7 The Rigby is published by Houghton Mifflin Harcourt. 
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the read-aloud. The student is also asked comprehension questions about the main idea of the 
story, primary characters, setting, and other salient features of the text. Oral fluency is assessed 
by asking the student to participate in a second read-aloud that is timed for one minute. Admin- 
istration of the Rigby PM assessment takes approximately 30 minutes or less depending on the 
student’s reading level. However, several iterations of the assessment may be administered to 
correctly identify a student’s independent reading level and to accurately place the student with- 
in the Reading Partners curriculum. 

The Rigby PM assessment is administered again midway through the school year, and 
once more at the end of the school year. Each student’s progress is documented in an IRP, a 
document that contains a student’s Rigby scores at the beginning, middle, and end of the school 
year; goals for the student; and areas to focus on. After the midyear Rigby assessment, Reading 
Partners staff use the data to update the IRP, identify areas for growth, and adjust student 
placement where needed. For students who are struggling or are not making gains during the 
first half of the school year, a targeted action plan is developed. Students’ Rigby assessment 
data and progress are also shared with teachers and principals; some teachers also share these 
data with parents when report cards are distributed. 

Rigorous and Ongoing Training 

Reading Partners staff and AmeriCorps members undergo formal, instructional training 
before they are placed at a school. Fonnal training for Reading Partners staff takes two forms: 
organized training sessions held before the school year starts, and ongoing (usually monthly) 
sessions held throughout the school year. New program managers receive two weeks of training 
that provides an introduction to the Reading Partners program model, its history, core instruc- 
tional beliefs, and culture, as well as rigorous training on the curriculum and other systems, such 
as how to manage internal databases throughout the program year. Experienced program man- 
agers and all site coordinators (both new and returning) also undergo intensive training before 
each school year begins. Topics that are covered during these training sessions include a de- 
tailed review of the curriculum, how to set up a Reading Partners reading center, how to train 
volunteer tutors, and how to use data to inform instruction. The ongoing sessions for Reading 
Partners staff, which are provided by more senior Reading Partners staff, are offered on specific 
topics and range from training that focuses on improving organization and relieving stress to 
how to better support beginning readers. 

For volunteer tutors, the primary training is a “shadow session,” which is a combina- 
tion of orientation and tutoring observation led by the site coordinator. The entire training 
takes less than one hour and then tutors begin working with their own students. Ongoing 
training sessions, led by site coordinators, are also available to volunteer tutors but are not 
mandated by Reading Partners. Topics range from how to better support English language 
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learners to managing students’ behavior and attention. These sessions are offered on a regular 
basis, often twice a month. 

Instructional Supervision and Support 

Reading Partners staff provide instructional supervision and support on an ongoing 
basis. Site coordinators provide supervision, coaching, and support to volunteer tutors, and pro- 
gram managers provide support and guidance to site coordinators. 

Program managers provide instructional supervision and support to site coordinators in 
two ways. First, program managers provide technical support related to implementing the Read- 
ing Partners model. In this role, program managers work with site coordinators to troubleshoot 
problems on a range of issues, including challenges related to specific students and their pro- 
gress or behavior, working with tutors and managing their performance and attendance, helping 
to manage communication with both teachers and administrators, and identifying best practices 
to better support a struggling reader. Second, program managers may also support site coordina- 
tors, who are often recent college graduates, by working with them on issues such as time man- 
agement and organization. Program managers have regular check-in meetings with site coordi- 
nators, but also provide infonnal, impromptu support on an as-needed basis. 

Site coordinators also provide tutors with instructional supervision and support on an 
ongoing basis. As managers of the reading center, site coordinators are there every day and ob- 
serve instruction as volunteer tutors work one-on-one with students. Site coordinators may pro- 
vide suggestions and guidance on what reading or engagement strategies to use with a particular 
student, and may also sit in on a tutoring session to provide additional instructional support. 

These aspects of the Reading Partners program are revisited in Chapter 3. That chapter 
includes an assessment of the degree to which the program was implemented with fidelity to the 
model across the study sites and a discussion of both the stronger aspects and the challenges of 
program implementation. 
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Chapter 3 

Understanding Implementation Fidelity 


The primary goal of exploring implementation and measuring fidelity in this study is to assess 
the extent to which the Reading Partners model is implemented as intended across all 19 
school sites. 1 Measuring the program’s adherence to the model helps the reader understand 
how the program impacts relate to implementation. Measuring fidelity also ensures that the 
outcomes from a well-designed impact study like a randomized controlled trial can in fact be 
attributed to the program model itself. 2 Finally, a thorough understanding of implementation 
fidelity can help readers understand whether the findings are generalizable to other Reading 
Partners locations. 

Specifically, this chapter explores three broad categories of implementation questions: 

• Context: What were the characteristics of the schools included in the study 
and the students served by Reading Partners in those schools? Who delivered 
the program, and how were they recruited and trained? How did the school 
setting affect program delivery? 

• Service delivery: Was the program delivered as intended — that is, with fi- 
delity to the model? What were the barriers to and facilitators of program 
delivery? Did particular aspects of the model pose special problems? Were 
there categories of schools with noticeably stronger or weaker implementa- 
tion, and did those differences affect program impact? 

• Implementation lessons: How did local contextual factors influence the im- 
plementation of the Reading Partners instructional model? What implemen- 
tation challenges were encountered and how were they addressed? What are 
lessons for other programs that may wish to implement similar strategies? 


Data Collection 

To collect the data necessary to answer these questions, in the winter of 2013, two-person teams 
made one-day visits to each of the participating sites to assess the context in which the program 


'Other implementation studies may also focus on other dimensions of fidelity, such as quality of imple- 
mentation and participant responsiveness (Dane and Schneider, 1998; Durlak and DuPre, 2008). Those dimen- 
sions were not a primary focus of this fidelity analysis. 

2 0 ’Donnell (2008). 
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was administered and to gauge the quality of program implementation. Seven members of the 
research team conducted these visits, with each member making between two and eight visits. 
Before they made their visits, the research team attended a half-day training session that includ- 
ed a review of all protocols, discussion of appropriate behavior in schools, and guidelines for 
conducting effective interviews. Written guidelines for the site visits were distributed before the 
training was conducted, which the research team members reviewed independently. 

The site visits included interviews with the program manager, site coordinator, outreach 
coordinator, regional executive director, school principal, school reading coordinator or reading 
specialist (if there was one), and approximately three tutors at each site. 3 Site visitors also con- 
ducted short, unstructured observations of three tutoring sessions and filled out a “facilities 
checklist” for each site, in which they recorded whether the room was set up as intended by 
Reading Partners and whether or not it had all required materials. Finally, the site visit included 
a systematic review of 10 randomly sampled student folders. The student folders contained in- 
fomiation about session attendance, the person who conducted the session, results of the 
assessments, the content of each tutoring session, and notes from tutors about what was covered 
in each session. An overview of the data that were collected as part of the implementation study 
is shown in Table 3.1, and copies of protocols from site visits are available upon request. 

To help augment the information collected during site visits, extant program data were 
obtained from Reading Partners’ own management information system (MIS). These data in- 
cluded information about individual student tutoring start and end dates, attendance, the number 
of different tutors assigned to a student over the course of the school year, and the background 
characteristics of both students and tutors. 

As described in more detail in Appendix A, interview transcripts, notes, and observa- 
tion write-ups were analyzed using a systematic coding process. A coding scheme was devel- 
oped to align with the overall framework of the implementation study and was organized to cap- 
ture data in the three main categories outlined above, and team members were assigned a set of 
codes to apply across all of the qualitative data. Once all data had been coded, team members 
provided a written synthesis that described emerging themes. 

The discussion of findings begins with an overall assessment of implementation fidelity 
as measured by a study-developed fidelity index, and is followed by a more in-depth discussion 
of each component. The chapter concludes with a discussion of the factors that affected the im- 
plementation fidelity of the Reading Partners program. 


3 Some interviews were conducted by phone when an in-person interview was not possible. 
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Reading Partners Evaluation 
Table 3.1 


Summary of Implementation Data Collection, by Source 


Data Source 

Number 

Targeted 

Number 

Collected 

Interviews 



Principal 

19 

19 

Reading coordinator 

11 

11 

Regional executive director 

6 

6 

Outreach coordinator 

7 

6 

Program manager 

10 

8 

Site coordinator 

17 

17 

Tutor 3 

51 

50 

Student folder reviews’ 5 

170 

171 

Reading center checklists 

17 

17 

Tutor observations 1 

51 

51 


SOURCE: Site visits to study schools. 

NOTES: a At two schools, only two tutor interviews were conducted. At one school, four 
tutor interviews were conducted. 

b Student folder reviews looked for documentation of student Rigby assessments, notes 
from tutoring sessions and on progress through the curriculum, and students' 

Individualized Reading Plans. At one site, 1 1 folders were reviewed. 

c At three schools, only two tutor observations were conducted. At one school, four tutor 
observations were conducted, and at one other school, five tutor observations were 
conducted. 


Implementation Fidelity Measure 

Implementation fidelity was assessed along the six core components of the Reading Partners 
model, as described in detail in Chapter 2: (1) regular, one-on-one tutoring; (2) dedicated school 
space and use of materials; (3) structured and individualized curriculum; (4) data-driven instruc- 
tion; (5) rigorous and ongoing training; and (6) instructional supervision and support. To help 
summarize and quantify the overall level of implementation fidelity across all 19 schools, the 
study team developed a fidelity index based on five of the six core components. The third com- 
ponent listed above — structured and individualized curriculum — was not incorporated into 
the index because the data collected were not appropriate for quantifying fidelity along this 
dimension. This component is discussed qualitatively. 
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The fidelity index contained 1 7 items that were created using quantitative data from the 
Reading Partners’ MIS; qualitative assessments of fidelity based on interviews with Reading 
Partners staff, school staff, and volunteer tutors; a review of student folders that are kept on site 
in reading centers; and a facilities checklist, which assessed whether the reading centers con- 
tained the required components. Criteria for each item were developed by the implementation 
study team in collaboration with Reading Partners. The respective weights given to each com- 
ponent of the fidelity index were determined by Reading Partners’ a priori assessment of the 
most important characteristics of the program and not necessarily what the research team con- 
cluded retroactively. For example, within the category of “dedicated school space and use of 
materials,” Reading Partners maintained that having a dedicated space that could not be shared 
was more important than having all required materials in place in the reading center. For this 
reason, having a dedicated space that did not have to be shared was weighted more heavily than 
having all required materials. Each Reading Partners school was scored across all 17 items. 
Two members of the research team and the project’s principal investigator reviewed all scoring 
decisions. Fidelity was assessed using data from the entire program year. A copy of the fidelity 
index, its key indicators, data sources, and criteria for scoring can be found in Appendix A. 

Overall, the fidelity index reveals that implementation fidelity across all 19 schools was 
relatively high, and that the program outcomes can likely be attributed to the Reading Partners 
program model. As shown in Figure 3.1, the fidelity scores for the 19 schools range from a low 
of 12 to a high of 22; the maximum possible score on the index is 23. Scores were grouped into 
three categories of implementation based on their overall distribution. Scores of 1 5 and below 
(less than 65 percent of the total points possible) were considered “low fidelity”; scores of 19 or 
higher (over 83 percent of the total possible) were considered “high fidelity”; and scores be- 
tween those delineations (15.5 to 18.5) were considered “moderate fidelity.” Of the 19 sites, two 
schools were designated as low fidelity; nine schools had moderate fidelity; and eight schools 
fell into the high-fidelity category. Chapter 4 explores the extent to which this variability in 
fidelity explains variation in program impacts. 

Regular, One-on-One Tutoring 

The Reading Partners model states that all students who participate in the program 
should receive one-on-one tutoring, as opposed to small-group support, which is more typical of 
literacy programs across the country. The one-on-one support is the main component of the 
program that sets Reading Partners apart from other supplemental services that are available to 
struggling readers. As one regional executive director described, “I think a lot of the other 
organizations that we come across provide literacy in different fomiats and usually it’s in small 
groups, so I think that the one-on-one piece is critical.” 
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Reading Partners Evaluation 
Figure 3.1 

Fidelity Scores of Study Schools 



ABCDEFGHI J KLMNOPQRS 

Study schools 3 


H Regular 1 : 1 tutoring | | Space and materials ^Data-driven instruction [TjTj Training [HUH Supervision and support 

(maximum score = 3) (maximum score = 5) (maximum score = 5) (maximum (maximum score = 7) 

score = 5) 

SOURCE: Site visits to study schools. 

NOTE: The fidelity index comprises metrics evaluating the five key quantifiable components of the Reading Partners model. The total maximum 
score is 23. A further explanation of the fidelity index and scoring guidelines can be found in Appendix A. 

a To prevent identification of individual schools or programs, site letter identifiers are not consistent across analyses. 







Analysis of the MIS data and site coordinator interviews revealed that except for one 
school, tutoring was exclusively one-on-one at all sites. At one school, students sometimes met 
with a pair of tutors, who worked with the student at the same time — a strategy employed by a 
site coordinator to help manage the large number of volunteer tutors. The site coordinator at this 
site noted that a partnership with a local high school had in the past led to an influx of volunteer 
tutors that caused a “chaotic” environment in the after-school tutoring sessions. In response, the 
site coordinator paired the high school tutors in order to reduce the number of pairs she had to 
monitor from six one-on-one tutoring sessions to three two-to-one tutoring sessions. Other sites 
did not seem to have this challenge. 

In addition to one-on-one instruction, the Reading Partners model has a frequency com- 
ponent. The program is designed to provide students with two 45 -minute tutoring sessions per 
week. As shown in Table 3.2, analysis of the MIS data reveals that, on average, students 
received approximately 1.5 sessions per week for 28 weeks (about three sessions every two 
weeks), which is slightly less than the model specifies. The MIS data indicate that students 
attended approximately 79 percent of the sessions that were available to them, after accounting 
for school closures and vacations. There was, however, variation in participation levels by 
school, with students in some sites maintaining an average of 1.8 sessions per week while those 
from others averaged just 1.1 sessions per week. Some site coordinators (4 of 17) noted that the 
students who did not receive tutoring twice a week were usually those who were absent from 
school. School attendance and the dosage of Reading Partners services that were received are 
correlated with one another, but the correlation is relatively low (around 0.18). Thus, lack of 
consistent attendance among volunteers also likely contributed to the problem of low frequency 
of participation in some instances. 1 

While one-on-one support and frequency of tutoring are explicit goals of the model, one 
of the implicit goals of Reading Partners is that a student will benefit from a supportive relation- 
ship with a caring adult as a result of working consistently with the same tutor. However, since 
tutor consistency is an implicit goal of the model, it was not factored into the fidelity index and 
is thus discussed only descriptively in this section. 

Tutor consistency can be explored along three different dimensions — whether students 
were typically assigned to work with one or two tutors each week, the duration of the tutoring 
relationship, and how consistently the assigned tutors appeared for their regularly scheduled 
sessions. The MIS data indicate that students did not typically work with the same tutor for both 
of their weekly sessions. As shown in Table 3.2, about three-fourths of primary tutors (76 


Hhe research team could not assess the correlation between tutor attendance and dosage because of limi- 
tations of the data. 
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Reading Partners Evaluation 
Table 3.2 


Average Student's Experience in Reading Partners 


Average Student Experience 

Program 

Group 

School-Level Averages 
Minimum Maximum 

Number of sessions per week 

1.55 

1.11 

1.76 

Length of participation in program (weeks) 

28.13 

24.24 

32.01 

Student attendance rate 2 (%) 

78.76 

55.75 

88.98 

Number of tutors assigned 

2.52 

1.67 

3.60 

Duration of each tutoring relationship (weeks) 

19.81 

11.20 

26.01 

Scheduled sessions per week with primary tutor 1 ’ (%) 
Scheduled once per week 

76.38 

39.58 

91.24 

Scheduled twice per week 

23.62 

8.76 

60.42 

Sample size 

594 




SOURCE: Reading Partners management information system data. 

NOTES: Calculations for this table used data for all program group students in the respondent sample. 
Sample sizes for individual outcomes may fall short of the report sample size because of missing or 
unusable data. 

a Student attendance is calculated as the number of sessions attended divided by the number of 
possible sessions, assuming two sessions every week that school was open after the student entered 
Reading Partners. 

b Among students' assigned tutors, primary tutors were those who had the longest assigned 
relationship with the student. 


percent) worked with their students only once a week, and the remaining one-fourth (24 per- 
cent) worked with their students twice a week. 2 One of the factors that made maintaining con- 
sistency with the same tutor difficult is that many tutors were available to volunteer only once a 
week, meaning that it was often not possible for a student to have the same tutor for both ses- 
sions each week. In the study schools, tutors generally came once a week — some only for one 
session, others for two sessions or more, working with a different student in each one. 3 


2 Primary tutors were those who had the longest assigned relationship with a student, among that student’s 
assigned tutors. 

3 The exception to this scenario was work-study students who were recruited from local colleges and uni- 
versities and a few paid tutors who worked in centers for more significant periods of time and on multiple days. 
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Although students were typically assigned to work with two different tutors each week, 
trust and relationship-building could still occur if tutors worked with the student for an extended 
period of time and appeared regularly and consistently for their sessions with students. As 
shown in Table 3.2, the typical duration of the tutor-student relationship was 19.8 weeks, or a 
little less than five months — a little longer than the one-semester commitment that Reading 
Partners asks tutors to make. Data regarding tutor attendance were not available from the MIS, 
so the study team conducted a review of randomly selected student folders (10 at each site) to 
explore the number of tutoring sessions and the number of unique tutors a student had over a 
two-week instructional period. During this time, students ideally would have received four 
tutoring sessions by either one or two individuals. 

The student folder review revealed that among students who received at least the full 
dosage of tutoring sessions (meaning four over the course of two weeks), more than half (56 
percent) worked with three tutors or more over the two-week period of time. Among students 
who received three sessions over the two weeks, approximately one-fourth (27 percent) worked 
with a different tutor on each occasion. These snapshot data suggest that tutor consistency was 
somewhat limited. 

As a result of tutor absences, both program managers and site coordinators did a fair 
amount of tutoring themselves. On average, site coordinators reported tutoring four or five 
times a week (about once a day), but at the high end, site coordinators reported tutoring three or 
four times a day. This is still a relatively small percentage of the total number of tutoring ses- 
sions offered during a given week (sites can accommodate up to 40 tutoring sessions in a given 
day), but it is still probably more than is ideal. While Reading Partners expects that site coordi- 
nators and program managers will do some tutoring, too much takes away from their ability to 
coach and provide guidance and support to the regular volunteer tutors. 

Dedicated School Space and Use of Materials 

All tutoring is designed to take place in a dedicated reading center within a school. Vis- 
its to each study site revealed that all but one of the sites had dedicated space. The one site that 
did not have dedicated space was operating in a library. The reading center was partly walled 
off from the larger part of the library by bookshelves, but the space included computer work 
stations for general school use. Despite this set-up, Reading Partners staff appeared to be mak- 
ing the best of the limited space and doing what they could to ensure that the environment was 
conducive to tutoring. All other sites had dedicated classrooms for the Reading Partners reading 
center, which were used by up to eight tutor-student pairs at a time. 

Within the dedicated classroom space, Reading Partners also has a set of materials 
(such as literacy-themed posters, signs, and other decorations) and required classroom compo- 
nents (for example, a “take-home reading” area) that each reading center should have in place. 
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During their site visits, the study team reviewed the space and materials against a checklist of 36 
items provided by Reading Partners. Box 3.1 presents a full list of the items in the Reading Cen- 
ter Materials Checklist. Of those 36 items, only 20 items were mandatory and incorporated into 
the fidelity index scoring. Of the 20 items that Reading Partners expects sites to have in place, 
only two sites had fewer than 19 items in place. Taken together, analysis of the space and mate- 
rials at each site suggests that this component of the Reading Partners model displayed a high 
level of implementation fidelity. 


Box 3.1 

Reading Center Materials 

8 work stations 

Site coordinator desk 

- Chairs and tables 

Student boxes 

- Pencils and erasers 

- Sight word ring 

- Post-it® notes 

- Student folder 

- Bookmarks 

- Tutor folder 

Alphabet strip 

- Word journal or reading/writing journal 

Additional supplies 

- Worksheets 

Bins and boxes for “Read Aloud” 

“Take-Home Reading” area 

Box for completed work 

- Book crates 

Bulletin boards for student work 

- Chart and stickers 

Clock 

- Reflection forms 

Curricula crates 

- “Take-Home Reading” sign 

“Read Aloud” library 

Tutoring schedule 

Resource table for volunteers 

United States map 

- Calendar 

Wall resources 

- Shadow session packets 

Whiteboards 

- Tutor information binder 

Word wall 

- Tutor resources 

World map 


Structured and Individualized Curriculum 

As described in Chapter 2, each lesson begins with a tutor reading aloud to the student 
from a literary or informational text that the child has selected. While reading, the tutor pauses 
to discuss the book’s content and vocabulary. The tutor read-aloud and vocabulary review is 
followed by the introduction of a new skill or concept, which the student is then asked to apply. 
Brief observations with students and interviews with tutors revealed that this approach is very 
closely aligned with the way in which the curriculum is implemented in practice. However, the 
study team did not conduct the tutor observations with the intention of monitoring the curricu- 
lum implementation in a detailed and fine-grained way. Observations were designed to provide 
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the research team with a “broad strokes” understanding of how the curriculum was operational- 
ized during a tutoring session. As such, this component of the model is not included in the fidel- 
ity index score. 

Overall, interviews with site coordinators and tutors, as well as brief, unstructured ob- 
servations of the tutoring sessions, suggest that the Reading Partners curriculum was imple- 
mented with a moderate to high level of fidelity across all sites. Tutors generally followed the 
steps outlined above. Some of the lesson components include scripted language that tutors can 
read verbatim. However, this is not a requirement; the scripts and outlines are available as 
guidelines. Tutors who are new sometimes rely on the scripts more heavily until they feel more 
comfortable with the curriculum. 

Tutors and site coordinators describe the beginning of a session as starting when a tutor 
picks up a student from the classroom, well before the actual tutoring begins. This suggests that 
relationship- and rapport-building is an important, implicit feature of the model. One site coor- 
dinator described the conversation and rapport being built during that time as a way “to prepare 
them [the students] to come to Reading Partners.” Often tutors ask students about their weekend 
or their day so far, which lays the groundwork for the one-on-one relationship that helps to 
facilitate the session. 

In observations, some variation was observed in the quality of delivery of tutoring. For 
example, occasionally a tutor did not pause for comprehension or vocabulary checks or did not 
do a good job of modeling fluent reading during the tutor read-aloud portion of the session. 
However, these observations were largely anecdotal. Site coordinators were also asked to com- 
ment on the quality of their tutors, but for the most part, site coordinators equated tutor quality 
with consistency and commitment. Tutors who came when they were supposed to and called or 
e-mailed if they were unable to make their scheduled time were generally considered high 
quality. Assessing the quality of the tutor-student interaction may be an area for future explora- 
tion within Reading Partners. 

Data-Driven Instruction 

Nearly all Reading Partners program staff and volunteer tutors are aware that data are 
integral to the Reading Partners instructional model. As described in Chapter 2, at the center of 
this data-driven instructional model is the use of the Rigby PM Benchmark assessment, which 
determines placement within the Reading Partners curriculum and assesses students’ progress 
by testing their performance at baseline before they begin the Reading Partners curriculum, their 
growth midway through the school year, and their status at the end of the school year. Analysis 
of the MIS data revealed that 95 percent of eligible students were in fact assessed using the 
Rigby all three times. Also, Reading Partners staff appeared to be doing a consistent job of shar- 
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ing these data with schools; the study team found that only one site failed to share data with the 
school principal. 

However, there was somewhat less evidence regarding the use of the Rigby data to in- 
form, customize, and update individual students’ Individualized Reading Plans. Although a 
majority of the site coordinators mentioned that they used the midyear Rigby assessment to 
either create an action plan or advance students, program managers and site coordinators pro- 
vided more limited infonnation about the specifics of how individual student progress was 
monitored and what was done with students who were not making progress. The use of student 
folders to document each tutoring session was another source of data that Reading Partners used 
to monitor student progress as well as particular successes and challenges that the students 
faced. While most of the student folders that were included in the folder review contained 
detailed documentation about student work during the tutoring sessions, student work was not 
documented in at least one instance in 10 sites over the course of the two-week review period. 

Rigorous and Ongoing Training 

The Reading Partners model requires that all Reading Partners school-based staff mem- 
bers undergo formal, instructional training before they are placed at a school site. Interviews 
with Reading Partners staff revealed that there was a high level of participation in the initial 
training provided to Reading Partners staff. Virtually all had completed a carefully designed 
training regimen. The only exceptions were two program managers who did not complete the 
initial, formal training because they were brought on mid-year to replace departing program 
managers. However, these program managers were able to briefly overlap with their predeces- 
sors and were able to receive on-the-job training and ongoing support from colleagues. 

Interviews with site coordinators also revealed that there was a high level of participa- 
tion in the ongoing training sessions, which occurred monthly. With hardly any exceptions, staff 
at all levels found the training to be comprehensive, useful, and successful in preparing them to 
carry out their roles effectively. 

For tutors, the key — and really sole — training they received before beginning their 
first tutoring session with a student was through “shadow sessions,” a combination of orienta- 
tion and tutoring observation led by the site coordinator at the school where the tutor was as- 
signed. Analysis of interviews with tutors revealed that the majority of tutors participated in a 
shadow session; only three volunteers did not participate in any shadow session before begin- 
ning to tutor. Those who did attend reported that these shadow sessions were adequate for 
orienting the volunteers to the Reading Partners reading center and curriculum, and equipped 
them to tutor students successfully. In fact, at least 80 percent of tutors who were interviewed 
indicated that they felt adequately trained for the role of tutor. 
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Ongoing training opportunities for tutors appear to have been plentiful, but few volun- 
teers availed themselves of those opportunities. Site coordinators and tutors alike reported that 
additional training sessions were offered, often twice monthly, and tutors reported often that 
they were aware of the training. These training sessions covered topics that site coordinators 
thought would be helpful to tutors, such as how to motivate students or how to improve tutoring 
skills. Still, almost half of the tutors (23 of 5 1) reported that they never attended any of the train- 
ing sessions. By far the most frequent reasons given for not attending were scheduling conflicts 
and other commitments. 

Instructional Supervision and Support 

For ongoing support, tutors relied almost exclusively on site coordinators and in gen- 
eral felt well supported by them. (Forty- two of 50 tutors who were interviewed indicated that 
they felt adequately or well supported.) Each Reading Partners center in the study sites had a 
regular site coordinator present and managing the classroom. Tutors received support from 
site coordinators in several different forms. First, there was ongoing monitoring and occa- 
sional direct involvement by the site coordinators during tutoring sessions. In addition, site 
coordinators held “check-outs” with tutors following a session, during which they reviewed 
how a session had gone. Third, site coordinators annotated student folders: Four tutors report- 
ed finding sticky notes regarding individual students, giving advice on how to proceed (for 
example, specific topics to emphasize or ways to address problems that students seemed to be 
having). By far the most common kind of support was student-specific. Tutors brought issues 
or problems they were having with specific students to site coordinators for a consult. These 
issues might be as simple as how to pronounce a certain word, a more complex problem (such 
as how to get across certain consonant or vowel sounds, or how to define a word or concept), 
or motivational challenges. 

Site coordinators’ descriptions of their own jobs suggest that there was some variability 
in the amount and quality of coaching and support they provided to tutors. Some site coordina- 
tors described their role as more of a monitoring role — ensuring that tutors knew their sched- 
ules and were following the curriculum, while others discussed providing advice and guidance 
to tutors on a broader range of issues. 

Broadly speaking, site coordinators, program managers, and outreach coordinators all 
indicated that they had adequate ongoing support and clear lines of accountability that worked 
for them. One challenge, however, was the lack of a program manager for two sites, which re- 
duced the level of supervision and support available to the site coordinators in those programs. 
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Factors That Facilitated Implementation 

A number of factors stand out as helping schools maintain high levels of overall implementa- 
tion fidelity. First, the structured and scripted nature of the curriculum enables volunteers with 
a variety of backgrounds to easily integrate into the program and deliver instruction without a 
lot of formal training. Second, the overall quality of the Reading Partners staff and Arneri- 
Corps members is quite high, particularly at the school level (program managers and site 
coordinators): Interviews with program managers and site coordinators revealed that they 
were dedicated and effective in their work on the whole. Most Reading Partners staff at the 
program manager or regional executive director levels have come to the organization with 
classroom teaching experience and a passion for the organization’s mission — a clear strength 
of the organization. The support of an effective organizational structure also helps ensure that 
a high level of quality is maintained among staff. 

In addition, the training provided to Reading Partners staff is strong and helps ensure 
that staff members know what is expected of them and that they have the appropriate skills and 
tools to carry out their jobs effectively. 

Certain contextual factors, such as Reading Partners’ relationship with teachers and 
administrators, have also helped maintain high levels of implementation fidelity. Through its 
school-level staff (program managers and site coordinators), Reading Partners maintains strong 
working relationships with principals and teachers. As a result, principals and teachers in the 
study schools reported generally being very supportive of the program, citing the one-on-one 
relationships that are fostered between students and tutors as one of its most helpful features. 


Barriers to Implementation 

While several factors facilitated effective program implementation, a number of other contex- 
tual factors posed barriers. The pull-out nature of the program was one such barrier. Although a 
majority of teachers and principals were very supportive of the Reading Partners program, in 
some schools, particularly those with many pull-out services in addition to Reading Partners, 
some teachers were resistant to having students pulled out from their classrooms for Reading 
Partners, particularly during instructional time. In such cases, site coordinators had to work 
harder to schedule tutoring sessions and had less flexibility in providing make-up sessions for 
students who missed a tutoring session because they were absent or a tutor was unavailable. 
Schools also varied in the space they were able to provide for the reading center to operate. 
While most were able to provide dedicated space, some classrooms were small, which meant 
that tutor-student pairs were spaced close together and the setting was noisier as a result. Small- 
er spaces also meant that fewer students could easily be tutored at one time. 
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Student attendance also played a role in program implementation. Student absences 
made it difficult to ensure that they received two 45 -minute sessions per week. A small num- 
ber of tutors also mentioned some difficulties working with a student population that had 
greater needs than average students, particularly English language learners or students with 
behavioral issues, but generally did not believe that these difficulties defined their overall 
tutoring experience. 

With respect to volunteer tutor recruitment, school location and characteristics of a re- 
gion were important factors that affected implementation of the program. Schools that were not 
close to public transportation or were located in neighborhoods that were perceived as unsafe 
had the hardest time with tutor recruitment. In addition, some regions had fewer opportunities 
for fundraising and less potential for community partnerships that lead to volunteerism. 

The biggest barrier that Reading Partners faced in implementing the program was main- 
taining consistent tutor attendance and retention. As described in Chapter 2, tutors are asked to 
make a one-semester commitment, but several site coordinators reported that many volunteers, 
particularly high school and college students, did not stay beyond that period of time, making it 
more difficult to establish a strong one-on-one relationship with the student. As already noted, 
the duration of the average tutoring relationship was 19.8 weeks — a little less than five 
months, or just over a semester. In one school, the average duration of the tutoring relationship 
was only 1 1 weeks. 

Furthermore, volunteers varied in their consistency and commitment. Site coordinators 
reported that tutors sometimes failed to report at their scheduled time and at times did not notify 
the site coordinator beforehand. This situation made it challenging to ensure that students 
received two sessions each week. However, there were structures in place to address these prob- 
lems. As noted earlier, a full-time employee was charged with recruitment on an ongoing basis, 
which helped to ensure an adequate supply of tutors to respond to the relatively high level of 
turnover. Reading Partners sometimes used substitute tutors if tutor absences could be anticipat- 
ed, and site coordinators or other tutors often provided make-up sessions on Friday to students 
who had missed a session earlier in the week. These two tactics helped ensure that tutor incon- 
sistency did not mean that a student necessarily missed sessions because a tutor was absent. Ad- 
ditionally, tutors are required to leave notes in students’ folders indicating what they worked on 
together and any problems that were encountered; because it is expected that many students will 
see two or more tutors regularly, this documentation is necessary to provide continuity of in- 
struction. Site coordinators also stay well infonned about the progress of the students in their 
centers and serve as a point of continuity for students who participate in the program. Chapter 4 
explores the impact of tutor consistency on student outcomes. 
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While these factors made implementation more complicated, the obstacles were not 
insunnountable and did not significantly reduce Reading Partners’ ability to implement the pro- 
gram with fidelity. In fact, overall, Reading Partners maintained an impressively high level of 
fidelity and its staff members were clearly able to overcome many of these challenges. These 
factors, however, may become more significant barriers for Reading Partners programs that 
may not be as mature or established as the program sites that participated in this study. As such, 
these programmatic and contextual factors should be examined more closely when implement- 
ing the Reading Partners model at newer, less established school sites and at sites that are work- 
ing to further strengthen existing programs. 


Summary 

The Reading Partners program was implemented with a moderate to high level of fidelity across 
each of the key dimensions of the program: (1) regular, one-on-one tutoring; (2) dedicated 
school space and use of materials; (3) structured and individualized curriculum; (4) data-driven 
instruction; (5) rigorous and ongoing training; and (6) instructional supervision and support. 

• On average, students received approximately 1 .5 tutoring sessions per week 
for 28 weeks in a dedicated space with appropriate materials. 

• Ninety-five percent of eligible students were assessed three times using the 
Rigby assessment, and Reading Partners staff consistently shared student 
data with schools. 

• Site coordinators, program managers, and outreach coordinators all indicated 
that they had adequate training for their roles as well as ongoing support in 
their work and clear lines of accountability. 

• Despite the limited initial training that tutors received, they generally report- 
ed that they felt adequately trained and supported in their role, though indi- 
viduals relied on support from the site coordinator to deal effectively with 
more challenging students. 

• The biggest barrier that Reading Partners faced in implementing the program 
was maintaining tutor attendance and retention. However, there were struc- 
tures in place to address these challenges and, as a result, most students were 
tutored on a regular basis throughout the year, even if the same person did 
not consistently deliver that tutoring. 
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Chapter 4 

The Impact of Reading Partners 


Because Reading Partners was not able to serve all students who would have qualified for the 
program, MDRC was able to employ a rigorous evaluation design, called a randomized con- 
trolled trial, to determine the magnitude of the effect — or impact — of Reading Partners. Ran- 
domly assigning students into a program group and a control group helps ensure that there are 
no systematic differences between the two groups of students when the study begins and that 
any positive effects can be causally attributed to the program with a high degree of confidence. 
The analysis of impacts was designed to answer the following key research question: 

• Do students who have been randomly assigned the opportunity to partici- 
pate in the Reading Partners program attain different reading proficiency 
levels from those of a similar group of students who were not offered this 
opportunity? 

The study also explored a set of secondary questions that were not considered key out- 
comes because they were more distal to the intervention, but that the study team was interested 
in exploring: 

• Do students in the program group exhibit different attention levels and dif- 
ferent behaviors in the classroom from the control group? 

• Do students in the program group complete their in-class assignments and 
homework at a different rate from students in the control group? 

The analyses also explored the effects of the program among different subgroups of 
students. The subgroups that were examined reflect five characteristics that are both theoreti- 
cally relevant and policy-relevant to reading instruction: 

• Do impacts differ for students entering the program with higher levels of 
pre-intervention reading achievement versus those with lower achievement 
levels? 

• Are impacts different for younger students (in grades 2 and 3) compared with 
older students (in grades 4 and 5)? 

• Do impacts differ for male versus female students? 

• Do impacts differ for students identified as English language learners com- 
pared with students who are fluent in English? 
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• Do impacts differ for students with prior exposure to the Reading Partners 
program compared with students who were exposed to Reading Partners 
tutoring for the first time during this study? 

To put the impact findings in context, the study team also explored questions related to 
service contrast, or the differences in the services received by students who were assigned to the 
program group versus students who were assigned to the control group: 

• What is the difference in academic support offered to students in the program 
and control groups? What was the resulting service difference in the amount 
of reading instruction on average between the program and control groups? 

If participating in the Reading Partners program did not increase the quality or fre- 
quency of the overall supplemental reading services that the program group received, either 
because the program group did not fully participate in the program or because the control 
group received other services of equal quality or intensity, then it is unlikely that an impact of 
the program would be observed, even if the Reading Partners program were of high quality. 

Finally, for exploratory purposes, this chapter includes an investigation of the ways in 
which the program impacts vary based on measures of fidelity and context. 


Methods 

Student Sample 

The study team used a systematic process to obtain and randomly assign the student 
sample at each school. At the start of the school year, schools and Reading Partners staff pro- 
vided the study team with a list of students they wished to refer to the Reading Partners pro- 
gram, who had parental permission to participate in the program and the study. These lists were 
developed by individual schools and were based on a combination of test score review and rec- 
ommendations by teachers or school staff (or both). Within two days of receiving the list, the 
study team randomly assigned students within schools (using a random number generator) to 
the program group that was eligible to participate in the Reading Partners program or to an “as 
is” control group that would not participate in Reading Partners but was eligible to receive any 
other supplemental reading services. A total of 1,265 students in the 19 participating schools 
were randomly assigned. Students were randomized within grade groups (grades 2 through 3 
and grades 4 through 5) to ensure that equal numbers of upper and lower elementary school stu- 
dents were represented in the sample, since one of the research questions examines the differen- 
tial impact of Reading Partners by grade level. Schools often approach reading instruction dif- 
ferently in these two sets of grades. In grades 2 and 3, reading instruction generally focuses on 
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learning to read, with more attention given to basic reading skills such as decoding, or the use of 
phonics to read written words correctly. In grades 4 and 5, the focus is on reading to leam, with 
greater emphasis on vocabulary and comprehension. 1 Equal grade-group sizes maximize the 
study’s ability to detect potential differential impacts between these groups. 

The final respondent sample (those who took at least one follow-up assessment) in- 
cludes 1,166 students (594 students in the program group and 572 in the control group). These 
totals represent response rates of 92.0 percent for the program group and 92.4 percent for the 
control group, which were not statistically significantly different from one another. Descriptive 
statistics for the final sample at baseline (that is, at the start of the study) are shown in Table 4. 1 . 
More than half of the sample was Hispanic (65 percent) and one-fifth of the students were black 
(19 percent). About half were officially designated as English language learners (55 percent) 
and over 90 percent of the students were eligible for free or reduced-price lunch. Aside from a 
slightly larger number of students in the program group who had previously participated in the 
program (32 percent compared with 26 percent of the control group), there were no statistically 
significant differences between the program and control groups on these variables. When the 
team performed a correction to account for the multiple characteristics that were analyzed at the 
same time, even this difference is not statistically significant. 2 

During the course of the study, 589 students across the 19 participating schools re- 
ceived over 25,000 tutoring sessions and over 19,000 hours of tutoring, administered by 901 
community volunteers. 

Data Collection 

Following random assignment in the fall of 2012, reading assessments (one group as- 
sessment and two individually administered assessments) were given to all students in the study. 
Assessments were administered by independent assessors hired by the study team. Assessors 
attended a two-day training program before beginning assessments. Baseline testing occurred as 
soon as possible after random assignment. Follow-up testing, using the same three assessments, 
was conducted in the spring of 2013, as close to the end of the school year as possible. In addi- 
tion, in the spring of 2013, teachers were asked to complete a survey about the academic behav- 
ior and performance of each student in their classrooms who was part of the study sample (in 
both the program and control groups) and to report on the full range of supplemental reading 
services that each student had received during the school year. A copy of the teacher survey is 


'dial I (1983). 

Corrections for multiple hypothesis testing were made using the Benjamini-Hochberg procedure (Benja- 
mini and Hochberg, 1995). Regardless, a full set of co variates was included in all analyses to control for any 
unobserved differences between the two groups at baseline and to improve the precision of the estimates. 
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Reading Partners Evaluation 
Table 4.1 

Baseline Characteristics of Program and Control Group Students 


Characteristic 

Program 

Group 

Control 

Group 

Difference 

P- Value 

Male (%) 

54.88 

54.50 

0.38 

0.899 

Ethnicity (%) 

Black 

19.39 

19.03 

0.37 

0.864 

Hispanic 

65.09 

65.27 

-0.18 

0.944 

Asian 

8.43 

9.09 

-0.66 

0.691 

White 

5.90 

5.40 

0.50 

0.709 

Other 

1.18 

1.02 

0.16 

0.856 

Special education (%) 

11.62 

10.49 

1.13 

0.550 

English language learner (%) 

55.21 

55.50 

-0.29 

0.916 

Eligible for free/reduced-price lunch (%) 

91.35 

90.22 

1.13 

0.540 

Prior Reading Partners participation (%) 

31.82 

25.75 

6.07 * 

0.074 

Average age (years) 

8.79 

8.76 

0.03 

0.413 

Overage for grade a (%) 

9.83 

8.40 

1.43 

0.408 

Average baseline achievement scores' 3 

Comprehension 

574.74 

573.49 

1.26 

0.544 

Sight word efficiency 

91.15 

90.38 

0.77 

0.328 

Fluency 

0.03 

0.00 

0.03 

0.573 

Students by grade level (%) 

Grade 2 

23.06 

25.29 

-2.23 

0.256 

Grade 3 

29.46 

27.23 

2.23 

0.256 

Grade 4 

24.58 

25.72 

-1.14 

0.572 

Grade 5 

22.90 

21.76 

1.14 

0.572 

Sample size 

594 

572 




SOURCES: MDRC calculations using study team-administered baseline SAT-10, TOWRE-2, and 
AIMSweb scores; district-provided demographic data; Reading Partners management information 
system data. 

NOTES: Calculations for this table used data for all students in the respondent sample. Sample sizes for 
individual outcomes may fall short of the reported sample sizes because of missing or unusable data. 

The model allows estimates to vary randomly across schools and controls for the random assignment 
block. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

a Students are classified as "overage" for a certain grade if they were the following specified ages on 
September 1, 2012: 8 or older for 2nd grade, 9 or older for 3rd grade, 10 or older for 4th grade, and 1 1 
or older for 5th grade. 

b The analyses for student reading achievement were conducted using scaled scores for the SAT- 10 
(comprehension) and the TOWRE-2 (sight word efficiency) and sample-normed scores for the 
AIMSweb (fluency). Sample-normed AIMSweb scores range between 0 and 1. 
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Reading Partners Evaluation 
Table 4.2 


Summary of Data Collection for Impact Analysis, by Source 



Number Randomly 
Assigned 

Number 

Collected 

Percentage 

Collected 


Data Source 

Program 

Group 

Control 

Group 

Program 

Group 

Control 

Group 

Program 

Group 

Control 

Group 

Baseline reading assessments 

Comprehension 

646 

619 

617 

591 

96 


95 

Sight word efficiency 

646 

619 

616 

592 

95 


96 

Fluency 

646 

619 

619 

592 

96 


96 

Follow-up reading assessments 

Comprehension 

646 

619 

579 

567 

90 


92 

Sight word efficiency 

646 

619 

585 

562 

91 


91 

Fluency 

646 

619 

589 

562 

91 


91 

T eacher survey 

646 

619 

593 

565 

92 


91 

Study year attendance 

646 

619 

615 

599 

95 


97 

Study year state test scores 

646 

619 

567 

550 

88 


89 


SOURCES: MDRC calculations from study team-administered baseline and follow-up SAT-10, 
TOWRE-2, and AlMSweb scores; study team-administered teacher survey; district-provided 
attendance and state achievement data. 

NOTE: Reading comprehension was assessed using the SAT-10; sight word efficiency was assessed 
using the TOWRE-2; fluency was assessed using the AlMSweb. 


included in Appendix B. A summary of the data collected on students is provided in Table 
4.2. In addition, participating districts supplied the research team with state test score data, 
attendance information, and the demographic characteristics of the individuals in the sample. 
General demographic information about the schools (school size, racial/ethnic composition, 
number of students receiving free or reduced-price lunch, and so forth) was extracted from the 
publicly available Common Core of Data (CCD), published by the National Center for Educa- 
tion Statistics. 3 


3 See http://nces.ed.gov/ccd. 
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Primary Outcomes 

The primary outcome of interest in this study is student reading proficiency, as meas- 
ured by test scores. The research team wanted to select a set of assessments that would measure 
proficiency in multiple domains of reading (described below) and were reliable, valid, well 
aligned with the curricular emphasis of Reading Partners, able to detect small improvements 
among lower-scoring students, and feasible to administer in a relatively short amount of time. 
After reviewing a range of possible reading assessments and talking with reading experts, the 
following assessments were selected: a group-administered reading comprehension test (the 
Stanford Achievement Test, 10th Edition, or SAT-10), an individually administered sight word 
efficiency assessment (the Test of Word Reading Efficiency, 2nd Edition, or TOWRE-2), and 
an individually administered oral reading fluency assessment (the AIMSweb). 

• The SAT-10 reading comprehension subtest is a 30-minute assessment that 
measures students’ comprehension across four different modes: initial under- 
standing, interpretation, critical analysis, and awareness and usage of reading 
strategies. Students read short passages and then answer questions about 
them. 4 

• The TOWRE-2 sight word efficiency subtest took approximately five 
minutes to administer. The sight word efficiency subtest is a measure of an 
individual’s ability to pronounce printed words accurately and fluently. It as- 
sesses the number of real words printed in vertical lists that an individual can 
accurately identify within 45 seconds. 5 

• The AIMSweb is an individually administered one-minute assessment of 
oral reading fluency, conducted by measuring accuracy and speed of reading. 
Students read a passage aloud for one minute and the assessor records omis- 
sions of words, misreading, and substitution of words. The count of the num- 
ber of words read correctly is used as the outcome. 6 

Secondary Outcomes 

Based on the Reading Partners program model, the study team also identified several 
secondary outcomes that were more distal to the intervention but still of interest. 


4 NCS Pearson, Inc. (2004). Kuder-Richardson 20 reliability on this measure ranged from 0.84 to 0.93. 

s Torgensen, Wagner, and Roshotte (2012). Alternate-form reliability on this measure ranged from 0.90 to 
0.92. 

6 Pearson Education, Inc. (2012). Alternate-form reliability on this measure ranged from 0.93 to 0.95. Split- 
half reliability also ranged from 0.93 to 0.95 and inter-rater reliabilities on this measure were 0.99. 
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• Academic behavior. One secondary outcome of interest was the impact of 
Reading Partners on students’ academic behavior (for example, attentiveness, 
assigmnent and homework completion, and disruptiveness). Four individual 
items were included on the teacher survey (reproduced in Appendix B) to as- 
sess each of these aspects of student behavior and were combined to create a 
single composite measure. 7 The possible score ranged from a low of 4 to a 
high of 16. 

• Teacher-rated academic performance. Teachers were also asked to assess 
each student’s perfomiance in reading, performance in math, and academic 
performance overall, relative to other students in the same school. The im- 
pact of the Reading Partners program on teachers’ ratings of student academ- 
ic perfonnance was explored separately for these three items. 

• Attendance. Finally, attendance data obtained from individual school 
districts were used to explore the impact of Reading Partners on student 
attendance. 

The difference between the outcomes of the program and control group students is re- 
lated to the service contrast, or the difference in reading services received by students in the 
program and control groups. Service contrast was measured using additional data provided 
through the teacher survey. 8 Teacher surveys asked teachers to identify how much time each 
student spent receiving in-class instruction in reading and how much of that total time was 
one-on-one instruction in class. Teachers were also asked to indicate any supplemental read- 
ing services, such as participating in pull-out reading instruction with a reading specialist that 
the student had received over the course of the school year and the amount of time spent in 
those supplemental activities. Reading Partners was included among the list of possible sup- 
plemental services. 

Analytic Approach 

The analysis pools together the sample of students across schools and compares out- 
comes for students who were assigned to participate in Reading Partners (the program group) 
with those who were assigned to the control group and thus did not receive Reading Partners’ 


7 The composite score was created by reverse-coding each of the items (except disruptiveness) and then 
summing the scores to obtain a single measure of academic behavior, with higher scores representing better 
composite behavior. Cronbach’s alpha for this composite measure, which assesses the coherence of the items 
within the measure, was 0.82. 

8 School-specific teacher surveys included questions about supplemental services offered to students at that 
school. These services were identified during the site visits that took place earlier in the school year. 
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services . 9 The model includes information about each student’s school and grade level to ac- 
count for the study design . 10 The model also controls for several baseline covariates to improve 
precision. These covariates include an individual-level pretest measure and the time lapse 
between baseline testing and follow-up testing, student’s gender, race/ethnicity, free or reduced- 
price lunch status, age, English language learner status, special education status, and whether 
the student is overage for grade. The complete model used in the analysis can be found in 
Appendix B. 

The same basic model is used to estimate the impact of the program on service contrast; 
however, in this case the outcome of interest was participation in all supplemental services 
(including Reading Partners) or time spent in reading instruction. 

As noted above, the analyses also explore the effects of the program among subgroups 
of students defined by characteristics depicting a student’s pre-random assigmnent condition, 
including grade level, baseline academic performance, gender, prior exposure to Reading Part- 
ners tutoring, and English language learner status, and the ways in which the program impacts 
vary based on measures of fidelity and context. Subgroup and other exploratory impacts — 
those that cannot be causally attributed — were estimated using only the members of a student- 
or school-level subgroup. 

The remainder of this chapter includes a description of the findings from the random 
assignment study. It begins by assessing whether or not the program had a statistically signifi- 
cant impact on the amount of instructional time in reading that the program group received 
and on their receipt of all supplemental reading services (including Reading Partners). Next, 
there is a description of the impact of the program on reading proficiency (measured by pri- 
mary outcomes) and on the secondary outcomes of interest. Finally, this chapter presents an 
exploration of whether the program was more effective for some subgroups of students than 
for others and the contextual and implementation factors that may have contributed to any 
variation in impacts. 


9 “lntent-to-treat” impact estimates are used to characterize the program’s impact, meaning that students 
are grouped based on whether they were assigned to participate in Reading Partners or not as opposed to 
whether they actually received Reading Partners services. 

10 The impact estimate is allowed to vary randomly across schools — this is considered a “random-effect” 
estimate of the program’s impact and can be generalized to the broader population of schools and students 
served by Reading Partners. 
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Understanding the Contrast Between the 
Program and Control Groups 

The design of this study compared students who were offered the opportunity to participate in 
the Reading Partners program with equally reading-challenged students who were not offered 
that same opportunity. However, since reading is such an important component of elementary 
education, it is unlikely that schools allowed struggling readers who were not assigned to the 
Reading Partners program to continue to struggle without providing some additional support. 
Moreover, given the limited resources available to schools, it is possible that once students were 
involved with Reading Partners, the school determined that they did not need other types of 
reading services as much. Thus, it is important to assess how different the program group’s ser- 
vices were compared with the services that the control group received. 

To assess this difference, teachers were invited to respond to a short survey about their 
students who were participating in the study. The survey, which was administered in the spring 
of the study year, asked teachers to report the amount of time students spent in classroom read- 
ing instruction each week and the supplemental reading services, including Reading Partners, 
that students who participated in the study received over the course of the school year. The find- 
ings from this survey are shown in Table 4.3. 

Time Spent in Reading Instruction 

The top panel of the table shows the amount of time that students in the program and 
control groups spent in classroom reading instruction each week, as well as the difference in the 
amount of instructional time between the two groups. On average, students in both the program 
and control groups spent around six hours each week on reading instruction in the classroom. 
Of that time, though not shown in the table, approximately 35 minutes was spent working with 
an instructor one-on-one. There was no statistically significant difference between the two 
groups in the total amount of in-class instructional time spent on reading instruction or in the 
time students spent working in class with an instructor one-on-one. 

However, the program group did spend more time receiving supplemental reading in- 
struction than did the control group. The Reading Partners group received about one more hour 
(57 minutes) of supplemental reading instruction per week than did the control group. This 
amount is less than would have been expected given the planned intensity of the Reading Part- 
ners program, which is designed to provide services to students for an hour and a half each 
week. The next section explores the source of this difference: whether students did not receive 
the intended dosage (frequency and intensity) of Reading Partners services, whether the stu- 
dents in the control group were also spending a substantial amount of time in other supple- 
mental services, or both. 
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Reading Partners Evaluation 
Table 4.3 


Reading Instruction Received 


Outcome 

Program 

Group 

Control 

Group 

Difference 


P-Value 

Average reading instruction time (weekly minutes) 
In class 

352.39 

360.29 

-7.89 


0.446 

In supplemental services 

177.66 

120.48 

57.18 

*** 

<0.001 

Students receiving any supplemental service (%) 

97.24 

64.75 

32.49 

*** 

<0.001 

Homework help 

11.21 

10.86 

0.35 


0.844 

One-on-one tutoring a 

95.17 

20.81 

74.36 

*** 

<0.001 

Small-group intervention support 

22.59 

31.53 

-8.94 

** 

0.014 

Technology-based programs 

8.28 

9.95 

-1.68 


0.279 

Other programs 

7.24 

7.48 

-0.23 


0.647 

Sample size 

580 

554 





SOURCES: MDRC calculations using study team-administered teacher survey; district-provided 
demographic data; study team-administered baseline SAT-10 scores; Reading Partners 
management information system data. 

NOTES: Calculations for this table used data for all students in the respondent sample with teacher 
survey data. Sample sizes for individual outcomes may fall short of the reported sample sizes 
because of missing or unusable data. 

The analysis model allows estimates to vary randomly across schools and controls for the 
random assignment block, the student-level pretest measure, the time lapse between baseline and 
follow-up testing, and student-level demographic covariates. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

a One-on-one tutoring includes the receipt of Reading Partners and other one-on-one 
supplemental services. 


Receipt of Supplemental Services 

To explore the difference between the program and control groups’ receipt of supple- 
mental services more closely, teachers’ survey responses on this subject were categorized into 
five types: 

1. Homework help: unstructured before- or after-school assistance and support 
focused on completion of teacher-assigned work 

2. One-on-one tutoring: Reading Partners and programs similar to Reading Partners, 
where the student worked one-on-one with an adult on reading-related activities 
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3. Small-group intervention support: usually a group-administered pull-out 
program for struggling readers, conducted by a school reading specialist or other 
educator 

4. Technology-based programs: those in which the student worked on improving 
literacy skills using a computer- or tablet-based application 

5. Other programs: reading services provided to students that did not fall into one of 
the above categories 

The bottom panel of Table 4.3 shows the impact of the Reading Partners program on 
the percentage of students receiving each type of supplemental service. The program had a posi- 
tive impact on receipt of total supplemental reading services overall. Ninety-seven percent of 
the program group students received some sort of supplemental reading instruction (including 
Reading Partners). However, 65 percent of the control group also received some type of sup- 
plemental reading instruction. Not surprisingly, the biggest difference was in the percentage of 
students receiving supplemental one-on-one tutoring, the category that included Reading Part- 
ners. Teachers reported that 95 percent of students in the program group were receiving sup- 
plemental one-on-one tutoring compared with 21 percent of the students in the control group. 
This table also shows that, in addition to Reading Partners, many students in the program group 
also received other supplemental services, though not always to the same extent as the control 
group students. Students in the program group were less likely to receive small-group interven- 
tion support than were students in the control group; 23 percent of program group students 
received this type of support as opposed to 32 percent of the control group. 

As shown in Figure 4.1, these findings suggest that through their participation in Read- 
ing Partners, program group students received around 50 minutes of additional instruction in 
reading each week relative to the control group, in large part because the control group also 
spent a substantial amount of time in supplemental instruction, particularly in small-group inter- 
vention support. As such, the impacts reported below should be interpreted as the impact of 
Reading Partners relative to other supplemental service receipt, not the impact of Reading Part- 
ners compared with no intervention. 


Impact of Reading Partners on Measures of Reading Proficiency 

The results shown in Table 4.4 represent the primary outcomes of interest in this study — 
namely, the impact of the program on reading comprehension, sight word efficiency, and fluen- 
cy. They show that overall the program had a positive and statistically significant impact on 
all three measures of students’ reading proficiency, with effect sizes equal to 0.10 for reading 
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Reading Partners Evaluation 
Figure 4.1 

Time Spent in Reading Instruction and Supplemental Services 
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SOURCES: MDRC calculations using study team-administered teacher survey; district- 
provided demographic data; study team-administered baseline SAT- 10 scores; Reading 
Partners management information system data. 

NOTES: Calculations for this table used data for all students in the respondent sample 
with teacher survey data. Sample sizes for individual outcomes may fall short of the 
reported sample sizes because of missing or unusable data. 

The analysis model allows estimates to vary randomly across schools and controls for 
the random assignment block, the student-level pretest measure, the time lapse between 
baseline and follow-up testing, and student-level demographic covariates. 

The sample size for the program group is 580; the sample size for the control group is 
554. 

Rounding may cause slight discrepancies between the numbers shown here and those 
shown in Table 4.1. 
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Reading Partners Evaluation 
Table 4.4 

Primary Impacts of Reading Partners on Reading Proficiency 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Impact 
Effect Size 

P-Value 

Comprehension 

592.42 

588.94 

3.48 ** 

1.71 

0.10 

0.043 

Sight word efficiency 

92.78 

91.37 

1 42 *** 

0.50 

0.11 

0.004 

Fluency 

0.06 

-0.03 

0.09 ** 

0.04 

0.09 

0.031 

Sample size 

594 

572 






SOURCES: MDRC calculations using study team-administered baseline and follow-up SAT-10, 
TOWRE-2, and AIMSweb scores; district-provided demographic data; Reading Partners 
management information system data. 

NOTES: Calculations for this table used data for all students in the respondent sample. Sample 
sizes for individual outcomes may fall short of the reported sample sizes because of missing or 
unusable data. 

The impact analyses for student reading achievement were conducted using scaled scores for 
the SAT-10 (comprehension) and the TOWRE-2 (sight word efficiency) and sample-normed 
scores for the AIMSweb (fluency). Sample-normed AIMSweb scores range between 0 and 1. The 
model allows estimates to vary randomly across schools and controls for the random assignment 
block, the student-level pretest measure, the time lapse between baseline and follow-up testing, 
and student-level demographic covariates. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 


comprehension (measured by the SAT-10), 0.11 for sight word efficiency (measured by the 
TOWRE-2), and 0.09 for fluency (measured by the AIMSweb). 11 An explanation of effect sizes 
can be found in Box 4.1, but these impacts mean that at the end of the school year, students in 
the Reading Partners group were scoring, on average, two to three percentile points higher than 
were students in the control group on these assessments. For example, on the sight word reading 
assessment, the average student in the program group scored at the 36th percentile compared 
with the average student in the control group, who scored at the 33rd percentile. This difference 
is equivalent to approximately one and a half to two months of additional progress in reading 
relative to the control group. When looking at growth between the beginning and end of the 


11 All three impacts remain statistically significant after controlling for multiple hypothesis testing using 
the Benjamini-Hochberg procedure (Benjamini and Hochberg, 1995). 
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Box 4.1 

What Is an Effect Size? 


An effect size is a way of quantifying the size of the difference in outcomes between two 
groups (for example, a program group and a control group, or two different subgroups), or the 
“impact estimate.” Calculating an effect size, which is done by dividing the impact estimate by 
the standard deviation of the outcome measure, allows researchers to compare impacts across 
variables that are measured in different units. For example, increasing a test score by two 
points on a test that is scored on a scale of 1 to 5 points is much more meaningful than increas- 
ing a score by two points on a test that is scored on a scale of 1 to 1 00. One way to interpret the 
magnitude of the impact estimates is to use the following rule of thumb: Effect sizes of about 
0.20 or less are considered “small”; effect sizes of about 0.50 to 0.80 are considered “moder- 
ate”; and effect sizes of about 0.80 or more are considered “large.” 


‘Cohen (1988). 


school year on these three assessments, the evaluation finds that both groups scored higher at 
the end of the year than they did at the beginning of the year, but growth was greater for the 
program group than for the control group. (See Appendix Figure C.l.) 

To put these findings in context, it is helpful to think about the total amount of addition- 
al instruction students in the program group received. The typical Reading Partners student 
received around an hour of additional supplemental instruction each week for approximately 28 
weeks, or a total of 28 additional hours of supplemental instruction. 12 The typical student 
receives approximately 30 hours of classroom-based instruction in reading each month (assum- 
ing a 90-minute reading block each day). 13 Thus, Reading Partners provided students with about 
a month’s worth of reading instruction. Yet, as noted earlier, the program’s impact on reading 
achievement is equivalent to about one and a half to two months of learning. This means that 
students in the program group progressed a bit more than might have been expected given the 
amount of additional instruction time they received. 


12 The 48-minute incremental difference presented in Figure 4.1 takes into account the total difference in 
instructional time, including the small and non-statistically significant difference in the amount of whole-class 
instruction. As indicated in Table 4.3, the additional supplemental instruction is closer to one hour per week. 

°This is consistent with the amount of time teachers in this study reported that students spent in reading 
instruction. As shown in Table 4.3, teachers in this study reported that students in the control group received 
approximately 480 minutes of reading instruction each week (in-class plus supplemental), or 96 minutes a day. 
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Appendix C also presents exploratory analyses of the impact of Reading Partners on 
state assessment scores. No statistically significant impacts were found. 


Impact of Reading Partners on Academic Behavior, 

Performance, and Attendance 

The research team was also interested in understanding whether or not the program improved 
students’ academic behavior (for example, attentiveness in class) and overall academic perfor- 
mance in the classroom, which were secondary outcomes. Table 4.5 shows the impacts of the 
program on attendance and teacher ratings of school performance and academic behavior. The 
impacts were generally positive (with the exception of attendance) but not statistically signifi- 
cant. For example, teachers indicated that 73 percent of the students who were assigned to the 
program group completed their homework “usually” or “always” (shown as “Satisfactory 
homework completion” in the table), while only 70 percent of the control group students com- 
pleted their homework “usually” or “always,” but the difference was not statistically significant. 
Teachers also rated the academic performance of the students who participated in the Reading 
Partners program somewhat, though not significantly, higher than they did the control students’ 
perfonnance. For example, teachers indicated that 51 percent of the students in the program 
group were above average relative to their peers in overall academic perfonnance (“Satisfactory 
perfonnance overall” in the table) as opposed to 47 percent of the control group. 


Subgroup Analyses 

To explore whether or not Reading Partners was more effective for some groups of students 
than for others, the study team conducted subgroup analyses based on five prespecified charac- 
teristics of interest: baseline achievement (based on the fall reading comprehension assess- 
ment), 14 prior receipt of Reading Partners services, gender, English language learner status, and 
grade level. The results are shown in Table 4.6. These analyses suggest that the Reading Part- 
ners program is effective for a wide variety of students: Impacts did not differ significantly for 
students from different grade levels or baseline achievement levels, for male or female students, 
for those who were still learning English, or for those who had previously been served by the 
Reading Partners program. 

There are, however, a few patterns in these findings that are worth mentioning, although 
these patterns should be interpreted with caution since the subgroups are not statistically 


14 The SAT-10 was chosen as the basis for determining the baseline reading performance of the students 
because it is a more comprehensive assessment of reading proficiency than the TOWRE-2 or the AIMSweb. 
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Reading Partners Evaluation 
Table 4.5 


Secondary Impacts of Reading Partners on Students' Attendance 
and Teacher-Reported Achievement and Behavior 


Outcome 3 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Impact 
Effect Size 

P-Value 

Attendance (%) 
Academic outcomes (%) 

95.65 

95.83 

-0.18 

0.24 

-0.04 

0.466 

Satisfactory performance 
in reading 

43.55 

40.59 

2.97 

2.76 

0.06 

0.283 

Satisfactory performance 
in math 

55.40 

53.84 

1.56 

3.26 

0.03 

0.632 

Satisfactory performance 
overall 

51.40 

47.40 

4.00 

2.81 

0.08 

0.155 

Behavioral outcomes (%) 







Satisfactory homework 
completion 

72.96 

70.11 

2.85 

2.59 

0.06 

0.271 

Satisfactory assignment 
completion 

78.16 

76.59 

1.57 

2.41 

0.04 

0.515 

Satisfactory level 
of attentiveness 

65.16 

65.16 

64.82 

64.82 

0.34 

0.34 

2.78 

2.78 

0.01 

0.01 

0.901 

0.901 

Unsatisfactory level 
of disruptiveness 

9.71 

9.67 

0.03 

1.79 

0.00 

0.986 

Composite behavior 

12.49 

12.37 

0.12 

0.15 

0.04 

0.412 

Sample size 

594 

572 






SOURCES: MDRC calculations using study team-administered baseline SAT-10 scores; study team- 
administered teacher survey; district-provided demographic and attendance data; Reading Partners 
management information system data. 

NOTES: Calculations for this table used data for all students in the respondent sample. Sample sizes for 
individual outcomes may fall short of the reported sample sizes because of missing or unusable data. 

The analysis model allows program impact estimates to vary randomly across schools and controls 
for the random assignment block, the student-level pretest measure, the time lapse between baseline and 
follow-up testing, and student-level demographic covariates. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

a For academic outcomes, satisfactory performance corresponds to ratings of "Average," "Above 
Average," or "Far Above Average." For behavioral outcomes, satisfactory corresponds to "Usually" or 
"Always," except for disruptiveness, which is reverse-coded. 
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Reading Partners Evaluation 
Table 4.6 


Subgroup Analysis of Primary Impacts 




Comprehension 

Sight Word Efficiency 

Fluency 



Impact 

Impact 

Impact 


Sample 

Effect 

Effect 

Effect 

Subgroup 

Size 

Size P-Value 

Size P- Value 

Size P-Value 


Prior achievement 

1st quartile 

310 

0.02 

0.839 

0.22 ** 

0.030 

0.19 * 

0.080 

2nd quartile 

305 

0.16 * 

0.056 

0.08 

0.295 

0.05 

0.538 

3rd quartile 

291 

0.12 

0.258 

0.11 

0.276 

0.08 

0.348 

4th quartile 

260 

0.00 

0.975 

0.14 

0.106 

0.10 

0.300 

Bottom 3 quartiles 

906 

o n ** 

0.043 

0.10 ** 

0.022 

0.09 ** 

0.040 

Prior receipt of 
Reading Partners 

Prior receipt 

316 

0.11 

0.178 

0.05 

0.629 

0.03 

0.722 

No prior receipt 

812 

0.07 

0.142 

0.12 *** 

0.010 

0.11 * 

0.060 

Gender 

Male 

633 

0.08 

0.179 

0.13 ** 

0.014 

0.07 

0.212 

Female 

522 

0.11 * 

0.070 

0.09 

0.137 

0.14 ** 

0.023 

English language 
learner (ELL) status 

ELL 

634 

o n ** 

0.048 

o n ** 

0.041 

0.14 ** 

0.015 

Non-ELL 

503 

0.01 

0.845 

0.11 * 

0.061 

0.04 

0.577 

Grade level 

Grades 2 and 3 

605 

0.09 

0.132 

Q *** 

0.002 

0.09 

0.182 

Grades 4 and 5 

561 

0.13 * 

0.096 

0.05 

0.479 

0.10 

0.101 


SOURCES: MDRC calculations using study team-administered baseline and follow-up SAT- 10, 
TOWRE-2, and AIMSweb scores; district-provided demographic data; Reading Partners 
management information system data. 

NOTES: Calculations for this table used data for all students in the respective subgroups of the 
respondent sample. Sample sizes for individual outcomes may fall short of the reported sample sizes 
because of missing or unusable data. 

The impact analyses for student reading achievement were conducted using scaled scores for the 
SAT-10 (comprehension) and the TOWRE-2 (sight word efficiency) and sample-normed scores for 
the AIMSweb (fluency). The model allows program impact estimates to vary randomly across 
schools and controls for the random assignment block, the student-level pretest measure, the time 
lapse between baseline and follow-up testing, and student-level demographic covariates. 

A two-tailed t-test is used for all statistical tests presented in this table. Statistical significance 
levels between the program and control groups are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Differences between subgroups are not statistically significant. 
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significantly different from one another. First, the subgroup analyses based on the incoming 
reading achievement of the students in the study sample indicate that the sight word efficiency 
and oral reading fluency impact effect sizes were largest for students who performed in the low- 
est quartile of the study sample on the baseline reading comprehension test, with effect sizes 
equal to 0.22 and 0.19, respectively. In other words, the program group students who were low- 
est-performing at the beginning of the year performed the best on the sight word efficiency and 
fluency measures compared with their counterparts in the control group. Reading comprehen- 
sion impacts were largest for those in the second-lowest quartile, with an effect size equal to 
0.16. Although the impacts for these subgroups (that is, by reading achievement at baseline) 
were not statistically significantly different from one another, they do suggest that the program 
may have been more effective for the students who were lowest-performing at baseline. Fur- 
thermore, the impacts on reading comprehension are the largest for the middle two quartiles, 
suggesting that the program may have been more effective in improving the reading compre- 
hension skills of students after they had mastered the basic mechanics of decoding. 

To further investigate these findings, the study team explored the impact of the pro- 
gram on the percentage of students whose performance on the reading comprehension 
assessment (SAT- 10) at baseline placed them in the lowest quartile of reading achievement 
nationally. These students (who represent approximately 60 percent of the study sample) per- 
formed less well on the SAT-10 at the beginning of the study than 75 percent of their peers 
nationwide. The Reading Partners program had a statistically significant positive impact on 
the percentage of these students who had moved out of this category by the end of the study 
year. As shown in Table 4.7, at the end of the year, 19 percent of the program group students 
who scored in the bottom quartile nationally at baseline had moved out of the lowest quartile 
by the end of the year, as opposed to 12 percent of the control group. The team also explored 
the impact of the program on the percentage of students who moved up a national quartile on 
the SAT- 10 (including students from the lowest three quartiles who would thus be able to 
move up quartiles over the course of the year). There was no statistically significant impact on 
this outcome measure. 

Second, the program appears to be particularly effective for English language learners, 
with positive and statistically significant effects on all three reading assessments for this sub- 
group. Third, Reading Partners appears to have made a greater impact on reading comprehen- 
sion scores for students who had previously participated in Reading Partners, while it appears to 
have made a greater impact on sight word efficiency and fluency for those in their first year. 

Finally, in the study sample, Reading Partners appears to have had a greater impact on 
sight word efficiency among students in grades 2 and 3, while there was a greater impact on 
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Reading Partners Evaluation 
Table 4.7 


Impacts of Reading Partners on Students’ Reading Comprehension 
Relative to National Norms 



Sample Program Control 


Standard 

Impact 

Outcome 

Size 

Group 

Group 

Difference 

Error 

Effect Size P- Value 

Moved out of lowest 
national quartile on SAT- 10 
reading assessment (%) 

675 

19.16 

11.61 

7.55 ** 

3.07 

0.22 0.014 

Moved up a national 
quartile on SAT- 10 
reading assessment (%) 

1,094 

14.10 

10.71 

3.39 

2.50 

0.11 0.175 


SOURCES: MDRC calculations using study-administered baseline SAT- 10 scores; district-provided 
demographic data; Reading Partners management information system data. 

NOTES: Calculations for this table used data for all students in the respondent sample. The sample size 
for the first outcome includes only students who started the study year in the lowest national quartile, 
while the second outcome includes students who started the study year in the lowest 3 national quartiles. 

The impact analyses for student reading achievement were conducted using percentile scores for the 
SAT-10 (comprehension). The model allows estimates to vary randomly across schools and controls for 
the random assignment block, the student-level pretest measure, the time lapse between baseline and 
follow-up testing, and student-level demographic covariates. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 


reading comprehension for students in grades 4 and 5. This finding is consistent with the devel- 
opmental progression of reading proficiency and with the focus of reading in these grades. 15 
Again, these subgroup differences are not statistically significant and thus should be interpreted 
with caution. 


Exploring Variation in Outcomes Based on Fidelity and Context 

In an effort to understand the factors that influenced program outcomes, the study team also 
explored several potential sources of variation in program impacts: tutor consistency, tutor 
background, school-level fidelity, the number of years that Reading Partners had been in op- 
eration in the school, and the average number of sessions per week and per year in the school. 


15 Chall (1983). 
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All these factors were identified in the literature as potential sources of variability in the effec- 
tiveness of tutoring programs or were identified by Reading Partners as factors that were wor- 
thy of exploration. The research team hypothesized that tutor consistency, school-level fideli- 
ty, the number of years the Reading Partners program had been in operation at the site, and 
the dosage of services that students received would all be positively associated with out- 
comes. Previous literature also indicated that college students might be more effective than 
tutors with other backgrounds. 16 

These analyses are nonexperimental and are intended to be used for generating hypoth- 
eses but not for confirming proposed theories. Unlike the student subgroups discussed earlier, 
factors like tutor consistency can be influenced by the intervention and, as a result, they cannot 
be interpreted as causal. The findings from these exploratory analyses are shown in Table 4.8. 

Tutor Consistency 

First, to ascertain whether tutor consistency was related to outcomes, schools were cat- 
egorized into three equal groups of “fewest,” “moderate,” and “most” consistency based on the 
average number of tutors assigned to each student over the course of the school year, with fewer 
tutors per student reflecting higher consistency. The average number of tutors assigned per stu- 
dent in each school ranged from a high of 3.6 to a low of 1.7. There are no discernible patterns 
in the results. 

However, the average number of tutors assigned per student at the school level may not 
be the most appropriate measure of consistency. First, this is a measure of school-level con- 
sistency, and does not represent the experience that an individual student may have had. There 
was considerable variation within schools in the number of tutors to whom individual students 
were assigned. In fact, the variation among students within each of the study schools accounted 
for 84 percent of the total variation in the number of tutors assigned per student in the study 
sample, while the variation between school-level averages accounted for only 16 percent. Thus, 
most of the variation occurred within schools, not across schools. This means that a school-level 
categorization is likely a poor predictor of a student’s experience. 

Therefore, another analysis explored whether growth in reading proficiency varied 
based on the number of tutors to whom an individual student was assigned. (See Appendix Ta- 
ble C.3.) These analyses are correlational because there is no control group to use as a compari- 
son. As an example, tutors who were working with difFicult-to-teach students may have been 
more likely to stop volunteering. Nonetheless, there is no statistically significant relationship 


16 Elbaum, Vaughn, Hughes, and Moody (2000). 
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Reading Partners Evaluation 
Table 4.8 

Subgroup Analysis of Primary Outcomes 
Based on Fidelity and Context Measures 




Comprehension 

Sight Word Efficiency 

Fluency 


School-Level 

Subgroup 

Sample 

Size 

Impact 

Effect 

Size 

P- Value 

Impact 

Effect 

Size P- Value 

Impact 

Effect 

Size P-Value 

Average number of 
tutors assigned 3 

Fewest 

311 

0.17 

0.302 

0.10 0.281 

0.09 

0.254 

Moderate 

500 

0.08 

0.167 

0.11 ** 0.045 

0.08 

0.332 

Most 

355 

0.10 

0.217 

0.12 * 0.099 

0.09 

0.249 

Fidelity of 
implementation 11 

Low 

137 

-0.07 

0.546 

0.10 0.620 

0.14 

0.472 

Moderate 

469 

0.13 

0.129 

0.15 *** 0.008 

0.01 

0.883 

High 

560 

0.10 

0.138 

0.07 0.211 

0.12 ** 

0.034 


Years of Reading 


Partners operations 


Less than 2 years 

493 

0.17 

0.129 

0.08 

0.185 

0.01 

0.873 

2 years or more 

673 

0.07 

0.164 

0.12 ** 

0.018 

0.15 ** 

0.011 

Dosage in sessions 
per week 3 

Low 

347 

0.09 

0.440 

0.11 

0.149 

0.02 

0.760 

Moderate 

442 

0.16 ** 

0.039 

0.12 * 

0.088 

0.07 

0.313 

High 

377 

0.03 

0.732 

0.08 

0.234 

0.16 

0.138 

Dosage in total 
sessions 0 

Low 

354 

0.04 

0.688 

0.12 

0.162 

-0.02 

0.774 

Moderate 

464 

0.21 *** 

0.001 

0.15 ** 

0.015 

q *** 

0.008 

High 

348 

0.01 

0.893 

0.02 

0.778 

0.06 

0.448 


SOURCES: MDRC calculations using study team-administered baseline and follow-up SAT-10, 
TOWRE-2, and AIMSweb scores; district-provided demographic data; Reading Partners 
management information system data. 

NOTES: Calculations for this table used data for all students in the respondent sample. Sample sizes 
for individual outcomes may fall short of the reported sample sizes because of missing or unusable 
data. 
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Table 4.8 (continued) 


The impact analyses for student reading achievement were conducted using scaled scores for the 
SAT-10 (comprehension) and the TOWRE-2 (sight word efficiency) and sample-normed scores for 
the AIMSweb (fluency). The model allows estimates to vary randomly across schools and controls 
for the random assignment block, the student-level pretest measure, the time lapse between baseline 
and follow-up testing, and student-level demographic covariates. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels between program and control groups are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. Differences between subgroups are not statistically significant. 

a "Fewest" reflects the lowest tercile of average number of tutors assigned per student; "moderate" 
reflects the second tercile of average number of tutors assigned per student; "most" reflects the third 
tercile of average number of tutors assigned per student. 

b Low-fidelity schools are those with a fidelity score of 15.0 or lower; moderate-fidelity schools 
are those with a fidelity score between 15.5 and 18.5; high-fidelity schools are those with a fidelity 
score of 19.0 or greater. 

c Low-dosage schools are those in the lowest tercile of average sessions attended per student; 
moderate-dosage schools are in the second tercile of average sessions attended per student; and high- 
dosage schools are in the third tercile of average sessions attended per student. 


between the number of tutors assigned to a student over the course of the school year and 
growth in reading proficiency on any of the three measures. 

The number of tutors assigned to a particular student also does not provide any indica- 
tion of how consistently the assigned tutor attended the tutoring sessions. If a student were as- 
signed to only one tutor over the course of the year, but that tutor was consistently absent, that 
might be a less effective tutoring arrangement than a student who was assigned to two tutors 
who each came on a regular basis. Unfortunately, Reading Partners does not collect data on tu- 
tor absences, so this possibility cannot be explored directly. However, data from the student 
folders that were reviewed as part of the implementation site visits can provide a measure of the 
number of tutors an individual student saw over a two- week period of time. To explore this 
question, the team calculated the average number of sessions per tutor for a random sample of 
students over the two-week period. This number was then averaged to the school level and an 
analysis explored whether impacts varied across schools based on this calculated measure of 
tutor reliability. The results are shown in Appendix Table C.4. Again, there is no consistent pat- 
tern in the data. 

Tutor Background 

The background of students’ primary tutors and the associated student gains on the 
three reading assessments used in this study are shown in Appendix Table C.5. The primary 
tutor is defined as the tutor with whom the student was officially paired over the longest period 
of time. Although these analyses are only exploratory, among the sample there is no clear indi- 
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cation that the background of a student’s primary tutor has any effect on that student’s increased 
reading abilities. In contrast with prior research, student reading gains for individuals who are 
tutored by college or graduate students are not noticeably larger than those for volunteers with 
different backgrounds. 17 While these students had relatively high gains on the SAT- 10 (reading 
comprehension), their gains on the TOWRE-2 (sight word efficiency) and AIMSweb (fluency) 
are lower than the gains for many other groups. Additionally, the differences in test score 
growth across the background characteristics are not statistically significantly different from one 
another, for either reading comprehension or fluency. There is a statistically significant differ- 
ence in the sight word efficiency scores, suggesting that high school students, working adults, 
and retired individuals may have been more effective than college or graduate students in im- 
proving students’ sight word efficiency. 

Fidelity of Implementation 

The next analyses explored whether impacts varied based on the fidelity scores of the 
schools in the study. As described in Chapter 3, the study team created a fidelity index based on 
the core components of the Reading Partners model, to assess how well the program was being 
implemented across sites. The scores on this fidelity index were used to categorize sites. Low- 
fidelity schools were those receiving a score of 15.0 or lower on the fidelity index; moderate- 
fidelity schools received a score of 15.5 to 18.5; and high-fidelity schools received a score of 
19.0 or greater. Again, there is no consistent pattern in student outcomes based on the fidelity 
scores of the various sites. This may be because the program was implemented with relatively 
high fidelity across all the sites. 

Years of Operation 

There were also no consistent patterns based on the number of years that Reading Part- 
ners had been operating in a school. Sites that had been in operation for two years or more had 
larger effect sizes on both the sight word efficiency and fluency assessments than did schools 
that had been operating for less than two years, but the effect sizes were larger for reading com- 
prehension for the newer sites. Again, these differences are not statistically significantly differ- 
ent from one another. Since the pattern of results varies across outcomes and in some cases the 
differences are relatively small, this pattern of results likely occurred by chance. 


17 Elbaum, Vaughn, Hughes, and Moody (2000). 
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Dosage 

Finally, schools were categorized based on the average number of tutoring sessions that 
students received each week, which ranged from a low of 1.1 to a high of 1.8 across the 19 
schools. Schools were divided into three equal groups, low, moderate, and high. No consistent 
patterns were found and the differences across the subgroups were not statistically significant. 
Similarly, looking at the average number of tutoring sessions offered over the entire year, which 
combines both the duration and frequency of the tutoring that students received, there is no evi- 
dence that high-dosage schools are more effective. In fact, the schools with moderate dosage 
appear to be somewhat more effective than the schools with the highest levels of dosage, though 
the difference is not statistically significant. It is not clear why moderate-dosage schools would 
be more effective than high-dosage schools. To put this finding in context, however, it should 
be noted that overall the dosage of Reading Partners was quite high. As noted in Chapter 3, on 
average, students in the program group attended 79 percent of the tutoring sessions that were 
available to them (once school closures and holidays were accounted for). 


Summary 

In summary, in this evaluation the Reading Partners program had a positive and statistically 
significant impact on the reading proficiency of the students who participated in the program. 

• Reading Partners had a positive and statistically significant impact on all 
three measures of student reading proficiency, with effect sizes of 0.10 on 
reading comprehension scores, 0.09 on reading fluency scores, and 0. 1 1 on 
sight word efficiency scores. This is equivalent to approximately one and a 
half to two months of additional progress in reading relative to the control 
group. 

• Since students in the control group were also receiving supplemental reading 
services, these impacts should be interpreted as the impact of Reading Part- 
ners relative to other supplemental services receipt, not the impact of Read- 
ing Partners compared with no intervention. 

• The impacts were equivalent to what a student leams in approximately one 
and a half to two months of additional school and are therefore somewhat 
larger than might have been expected given the amount of additional instruc- 
tional time that students in the program received — 28 hours, or approxi- 
mately one month. 
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• Impacts did not differ significantly for students from different grade levels, 
for male or female students, for those who were still learning English, or for 
those who had previously been served by the Reading Partners program. 

• Reading Partners had a positive and statistically significant impact on the 
percentage of students who moved out of the lowest quartile nationally in 
tenns of reading comprehension. At the end of the year, 19 percent of the 
program group students who had scored in the bottom quartile nationally at 
the start of the study had moved out of the lowest quartile, as opposed to only 
12 percent of the control group. 

These findings all suggest that the Reading Partners model “works” — that is, it pro- 
duces measurable impacts in reading skills among participants. The findings indicate that the 
impacts may have been largest for those readers who struggled the most; however, the findings 
also indicate that the program was effective in helping students with a fairly broad range of 
reading abilities and from a variety of grade levels and backgrounds. Reading Partners produced 
these measurable and consistent impacts despite the lack of prior experience among tutors, the 
somewhat limited training they received, and the relatively short duration of the tutor-student 
relationships. Chapter 6 includes a discussion of the components of the Reading Partners model 
and which components may have contributed to the impact findings presented in this chapter. 
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Chapter 5 

The Cost of Reading Partners 


Cost studies in education provide important information to aid the policymaking process. 1 They 
often take one of three fonns: (1) a benefit-cost analysis in which the outcomes of a program are 
expressed as monetary benefits and related to the costs to determine whether the benefits of the 
program outweigh the costs; (2) a cost-effectiveness study that compares the relative costs and 
effects of two or more programs that target equivalent outcomes; and (3) a cost study that 
describes the costs of a program, based on a comprehensive list of the resources (or “ingredi- 
ents”) used to implement the program, and analyzes which entities bear the burden of financing 
those costs. 2 Since reading achievement is not easily expressed in dollars, a benefit-cost analysis 
could not be easily undertaken for this study. Similarly, a cost-effectiveness analysis could not 
be conducted because the evaluation did not include an alternative program with which Reading 
Partners could be directly compared. Therefore, this evaluation takes the third approach and 
provides a rich description of the costs of Reading Partners, as well as the costs of the other 
supplemental services available in a sample of the evaluation schools. 

Two types of “costs” are examined in this chapter. The first is the value of all goods and 
services used to implement the Reading Partners program, including in-kind (such as the class- 
room space and the time school staff spend selecting students for the program) or donated 
resources. This is what is often referred to as the “total cost” of the program. 3 The total cost is 
useful for thinking about the resource richness of the program and when comparing the 
resources used to implement various programs. The second type of cost discussed is the direct 
cost of the program borne by the school (that is, the portion of the total costs financed by the 
school or school district). Since principals, superintendents, or school boards are the agents 
making the decision to bring Reading Partners into a school, the cost the school bears is rele- 
vant. To calculate cost, the ingredients method is used. This method is a rigorous approach to 
conducting cost studies that accounts for all resources that are used to achieve educational 
gains. 4 The full resource cost, or the “total cost” to society, includes the value of all the ingredi- 
ents, regardless of who paid for or contributed them. 5 The cost borne by schools includes 
the value of the ingredients that the schools provided — both in-kind and financial. (Not all 
analyses of costs include the cost of in-kind contributions. Therefore, comparisons of the costs 

Hummel -Rossi and Ashdown (2002); Levin and Belfield (2013); Levin and McEwan (2001); Ross, 
Barkaoui, and Scott (2007). 

2 Levin (1975); Levin and McEwan (2001). 

3 Levin and McEwan (2001). 

4 Dhaliwal, Duflo, Glennerster, and Tulloch (2012); Harris (2008); McEwan (2002); Rice (1997). 

5 Levin (1975). 
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presented in this chapter with those in other reports will not be meaningful unless the ingredi- 
ents covered by the cost estimates are similar.) 

This chapter provides information about the resources needed to replicate the Reading 
Partners program at the level it was implemented in the study sites, and documents both the 
financial and in-kind resources that a school typically must provide to implement Reading 
Partners. In addition to presenting overall costs averaged across sites, the chapter covers how 
and why resources varied across sites. Studies have shown the importance of examining site- 
level variability in costs because understanding the pattern of resource use across sites may be 
important for evaluating a particular replication plan or for achieving a particular effect. 6 

As discussed in Chapter 4, an evaluation of Reading Partners is incomplete without 
also examining the other supplemental reading services provided in the schools, because the 
impacts presented in Chapter 4 are the impacts of Reading Partners relative to an environment 
that already has other such programs. Thus, in addition to assessing the total resources needed 
to implement the Reading Partners program, this cost study was designed to provide site-level 
estimates of the average cost of the other supplemental services provided at each school in the 
cost study sample. Each site had between one and four other supplemental reading services 
included in the study. The services included tutoring, read-aloud programs, reading special- 
ists, and computer-based programs. Some of these programs also used volunteers. The cost 
estimates for these other supplemental reading programs provide valuable information about 
the resource richness of Reading Partners in relation to the menu of other services that are 
typically provided for struggling readers. 

Specifically, this chapter addresses the following research questions: 

1. What resources are needed to implement the Reading Partners program as 
described in this evaluation and what proportion of the costs of implementing 
the program are borne by the school? 

2. What resources are needed to implement the other supplemental services 
available to struggling readers at a sample of sites that were included in the 
impact evaluation? 


6 Bowden (2014); Hollands et al. (2014); Levin, Catlin, and Elson (2007). 
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Methods 


The costs of Reading Partners and of the other supplemental reading services were estimated in 
six schools using the ingredients method, described below. 7 

Cost Study Sample 

Six school sites were selected from the larger evaluation for inclusion in the cost study. 
The team selected sites that had strongly implemented programs, that were geographically rep- 
resentative of the other sites in the study, and where reliable data could be collected on the re- 
sources used in Reading Partners and other supplemental reading services offered during the 
2012-2013 school year. The sites were all elementary schools that varied in the size of the 
school (that is, the number of students enrolled) and in the number of students served by Read- 
ing Partners. Table 5.1 shows the characteristics of the cost study sample compared with the 
other Reading Partners schools included in the evaluation. The schools in the cost study had 
more students enrolled and were composed of slightly more white students than the rest of the 
evaluation sample but were otherwise comparable. 8 In addition, the average impact on reading 
proficiency among these six sites was comparable with the full sample impacts, with effect sizes 
equal to 0.09 on each of the three reading proficiency measures. 

Data Collection 

All of the resources that were used to implement Reading Partners were identified from 
discussions with program administrators, interviews with and surveys of program and school 
staff, and data collected during the observational school site visits. The resources included all 
in-kind or donated ingredients such as the time contributed by the principal or the time contrib- 
uted by community volunteer tutors. Even though Reading Partners enlists tutors who volunteer 
at no cost to the school, the volunteers are a necessary and important ingredient for the program 
to work. To properly document the full resource richness of the program, a value is given to 
these donated hours. They are valued at the price that their services would cost if they had to be 
purchased. This full-cost valuation allows an appropriate comparison of the resource richness of 
programs that receive donated time with those that do not. 

Site by site, the study team compiled the full set of ingredients needed to replicate the 
program as it was implemented at the site. Because the focus of the study is on the cost of 


Additional information on the ingredients method can be found in Appendix D (Levin and McEwan, 
2001 ). 

8 This report does not reveal the total number of students served at each school or the number of students in 
the program group at each site to protect the identity of each site. 
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Reading Partners Evaluation 
Table 5.1 


Characteristics of Cost Study Schools 
and Other Reading Partners Study Schools (2011-2012) 


Characteristic 

Cost Study 
Schools 

Other 

Reading Partners 
Study Schools 

Difference 

P-Value 

Eligible for Title I program (%) 

83.33 

91.67 

-8.33 

0.621 

Students eligible for 

free/reduced-price lunch 11 (%) 

77.96 

83.43 

-5.47 

0.331 

Race/ethnicity (%) 

Black 

28.14 

17.32 

10.82 

0.337 

Hispanic 

49.67 

67.32 

-17.66 

0.120 

Asian 

10.34 

9.25 

1.08 

0.831 

White 

9.43 

3.98 

5.45 * 

0.072 

Other 

2.43 

2.12 

0.31 

0.798 

Male (%) 

51.93 

51.88 

0.05 

0.971 

Average number of students 

612.33 

439.62 

172.72 ** 

0.049 

Grade 2 

97.33 

70.15 

27.18 

0.158 

Grade 3 

93.00 

67.08 

25.92 

0.156 

Grade 4 

87.00 

68.08 

18.92 

0.276 

Grade 5 

88.83 

69.17 

19.67 

0.281 

School setting 11 (%) 

Urban 

66.67 

76.92 

-10.26 

0.659 

Suburban 

33.33 

23.08 

10.26 

0.659 

Town 

0.00 

0.00 

0.00 

NA 

Rural area 

0.00 

0.00 

0.00 

NA 

Sample size 

6 

13 




(continued) 


replication, or incremental costs, the costs of general program administration and develop- 
ment were not included. By separating cost by site, the study team was able to examine the 
range of resources that contributed to the operation of Reading Partners. 

Analytic Methods 

After a comprehensive list of ingredients was compiled, each ingredient was matched 
with its national price, or cost in 2012 dollars. National prices were obtained from publicly 
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Table 5.1 (continued) 


SOURCES: MDRC calculations from 2011 and 2012 National Center for Education Statistics 
Common Core of Data (CCD). 

NOTES: Sample sizes for individual outcomes may fall short of the reported sample sizes because of 
missing or unusable data. 

A two-tailed t-test is used for all statistical tests presented in this table. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

a The value given for students eligible for free/reduced-price lunch is calculated from the 2011 
CCD because data are missing in the 2012 CCD. Data for all other variables are from the 2012 CCD. 

b "Urban" is defined as territory inside an urbanized area and inside a principal city having a 
population greater than 100,000. "Suburban" is defined as territory outside of a principal city and 
inside an urbanized area with a population of less than 250,000. “Town” is defined as territory inside 
an urban cluster that is (1) less than or equal to 10 miles from an urbanized area (“fringe”), or (2) 
more than 10 miles and less than or equal to 35 miles from an urbanized area (“distant”), or (3) more 
than 35 miles from an urbanized area (“remote”). “Rural” is defined as territory that is (1) less than or 
equal to 5 miles from an urbanized area and less than or equal to 2.5 miles from an urban cluster 
("fringe”), or (2) more than 5 miles but less than or equal to 25 miles from an urbanized area and 
more than 2.5 miles but less than nor equal to 10 miles from an urban cluster (“distant”), or (3) more 
than 25 miles from an urbanized area and more than 10 miles from an urban cluster (“remote”). 


available sources such as the Bureau of Labor Statistics, the Department of Labor, the National 
Center for Education Statistics, and AmeriCorps. (More details are provided in Appendix D.) 

As already noted, volunteer time is a critical ingredient for Reading Partners, but it 
does not have a clear market rate. The price used must therefore be an approximate value of 
the ingredient based on similar resources with market prices. 9 The price reported here is the 
hourly rate for a paraeducator or teacher’s aide. 10 The role of the paraeducator and the role of 
the volunteer tutors (for both Reading Partners and the other supplemental reading programs 
that used volunteers) seemed most similar with regard to expectations and responsibilities. 
Thus, the full cost that is reported below represents the cost of the program including the costs 
reflected by the volunteers. This estimate could also be interpreted as the cost of these ser- 
vices if the tutors had to be paid. 

Each Reading Partners program site served up to 80 students. However, only a portion 
of those students served were included in the evaluation, either because eligible students were 
identified after randomization occurred or because the school served students in lower grade 
levels that were not included in the evaluation. The data collected reflect the services provided 


9 Levin and McEwan (2001). 

10 However, as described in Appendix D, sensitivity tests using minimum wage and tutor education levels 
provide lower and upper bounds for the price of the volunteers. 
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to all students who participated in Reading Partners and not just those in the evaluation sample. 
Therefore, the calculation of the cost per student must be adjusted to account for the additional 
students who were accommodated; otherwise, the cost per student would be overstated. The 
term “per program group student” is used throughout the report to refer to the students who 
were randomly assigned to participate in the Reading Partners program as part of the evaluation 
(the “program group”) and to distinguish them from the total pool of students served. 

To calculate the cost per program group student, first the total cost (or the total resource 
value) of the program at each site was divided by the total number of sessions provided to all 
students who received Reading Partners’ services during the year (not just the sessions provided 
to the study’s program group students) to obtain the cost per session. Then, to determine the 
average per-student cost of serving the program group, the average cost per session was multi- 
plied by the average number of sessions that program group students attended. 

The resources needed to run the Reading Partners program come from many different 
sources, including schools or school districts, volunteers from the community, the AmeriCorps 
program, and the Reading Partners organization. Therefore, the costs borne by each constituent 
are described below, with a special emphasis on the costs borne by the schools. When the costs 
to the school are reported, the estimate is based on the resources contributed by the school in- 
kind as well as the fee-for-service paid by the school to Reading Partners for the program. 

These Reading Partners costs are examined relative to the cost of the other supple- 
mental reading services provided in the school. This analysis shows the relative “richness,” or 
resource intensity, of Reading Partners, as well as the relative cost of Reading Partners to the 
school. The costs of the other supplemental reading services provided at each school were cal- 
culated in the same detailed way and followed the same pricing approach using national prices. 
The costs of the other supplemental programs were aggregated by averaging them at the site 
level to protect confidentiality. 


Findings 

The total resource value, or cost, of Reading Partners is approximately $3,610 per program 
group student, of which $1,910 reflects in-kind resource contributions made by schools and the 
community volunteers and the remaining $1,700 reflects the financial costs borne by the Read- 
ing Partners program, the AmeriCorps program, and the schools. (See Table 5.2.) As discussed 
in Chapter 6, other effective early literacy interventions that have been evaluated at scale are at 
least as costly as Reading Partners. 11 However, unlike many other resource-rich programs, a 


1 'Hollands et al. (2013); Simon (2011). 
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Reading Partners Evaluation 
Table 5.2 


Cost of Reading Partners per Program Group Student 


Cost 

Cost per 
Student 

Distribution of Cost per Student 
Cost to 

Cost to Cost to Reading Cost to 

School Volunteers Partners AmeriCorps 

Ingredients ($) 






Reading Partners staff 

690 



690 


AmeriCorps members 

930 



930 


School staff 

90 

90 




Volunteer time and transportation 

1,520 


1,520 



Facilities 

300 

300 




Materials and equipment 

80 



80 


Total ingredients 

3,610 

390 

1,520 

1,700 

0 

Fee for service 3 ($) 


320 


-320 


AmeriCorps grant 1 ’ ($) 




-270 

270 

Net cost per student (total ingredients + 






fee for service + AmeriCorps grant) ($) 


710 

1,520 

1,110 

270 

Portion of net cost per student (%) 


20 

42 

31 

7 


SOURCES: MDRC calculations from cost data. 

NOTES: Costs are measured in 2012 U.S. dollars and are rounded to the nearest 10 in the tables and 
text. 

Averages are weighted to account for the size of the program group at each site. 

All costs are those for the respondents in the program group and are adjusted for the larger dosage 
they received. 

Rounding may cause slight discrepancies in sums and differences. 

a Each school pays a fee to Reading Partners to finance a portion of resources provided by the 
organization, such as staff and materials. The fee presented here is the national average fee per student. 

b AmeriCorps provides a grant to Reading Partners to finance approximately 30 percent of the costs of 
the AmeriCorps members employed by the program. Reading Partners pays the remainder of the costs. 


majority of the costs of Reading Partners is covered by in-kind contributions. Furthermore, 
Reading Partners schools bear only a small portion of the total costs of the program. Because 
the program relies heavily on volunteers, on average schools bear only 20 percent ($710 per 
program group student) of the cost of the total resources required to implement the program, 
and over half of these costs are covered by in-kind contributions. The cost findings for Reading 
Partners are discussed in more detail below, along with the costs of the other supplemental read- 
ing services provided at the schools. 
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The School’s Portion of the Total Cost of Reading Partners 

While the total resource value of the Reading Partners program is $3,610, the schools 
themselves paid only a small proportion of these costs. As shown in Table 5.2, the schools in 
the study contributed both in-kind resources and a fee to finance the program. The in-kind re- 
sources included a classroom for the reading center, space for program events, and time from 
the principal, teachers, administrative staff, and the reading specialist or coordinator. The value 
of these in-kind resources is approximately $390 per program group student, of which $300 is 
the cost of the space. In addition to space and time, the schools also covered a portion of the 
resources provided by the Reading Partners sponsor by paying a fee of $320 per program group 
student to Reading Partners. Thus, the school’s commitment of $710 per student leveraged al- 
most $3,000 of additional resources, a majority of which is provided by the volunteer tutors. 

Volunteers’ Contribution to the Cost of Reading Partners 

In addition to the costs financed by the school, the remaining costs of the program were 
borne by three agents: the Reading Partners organization, the AmeriCorps program, and the 
volunteers themselves. 

As shown in Table 5.2, the largest proportion of the resources used in implementing the 
program (42 percent of the total) was financed through in-kind donations from the community 
volunteers. Volunteers contributed, on average, the equivalent of $1,520 per program group 
student, which included both their time and transportation costs, for which they were not reim- 
bursed. 12 The Reading Partners organization financed about 31 percent of the required resources 
($1,110 per program group student), which included both staff and materials. The site coordina- 
tor and outreach coordinator positions are partially financed by the AmeriCorps program. 
AmeriCorps provided a grant to Reading Partners to finance approximately 30 percent of the 
costs of the AmeriCorps members. Thus, AmeriCorps funding finances about 7 percent of the 
total cost of Reading Partners. 

In addition to considering the costs borne by each of the entities and the volunteers, it 
is also important to consider what proportion of the total costs were in-kind contributions. In 
the Reading Partners program, the resources contributed in-kind by the schools and communi- 
ty volunteer tutors totaled over 50 percent of the total cost, or $1,910 per program group stu- 
dent. The remaining $1,700 per program group student was financed by the Reading Partners 
organization and the AmeriCorps program, with some financial assistance from the schools 
themselves. 


12 Time and costs for travel are considered to be part of the in-kind donation made by the volunteers. The 
same approach was used for other programs that relied on volunteers. 
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Costs of Other Supplemental Reading Services 

In addition to Reading Partners, the six school sites that were included in the cost study 
offered other supplemental services that provided reading instruction beyond what students re- 
ceived during regular classroom teaching. In many cases, services were provided to students in 
need of additional support, irrespective of whether they had been assigned to the Reading Part- 
ners evaluation program group or control group. The costs of these other supplemental reading 
services are included here to provide a context for understanding the estimated level of resource 
richness (as measured by total cost) for Reading Partners. 

The other supplemental reading services included small-group support from reading 
specialists, tutoring programs, homework help, and computer-based programs. Table 5.3 shows 
the average cost per student for the other supplemental reading services at each school site. The 
costs of the other supplemental reading services are averaged at the site level to protect confi- 
dentiality, so that the cost per student represents the average cost per student of all supplemental 
programs at each school, weighted by the number of students served by each program. The 
number of services offered in each school ranged from one to four services in addition to Read- 
ing Partners. Some of these averages combine relatively resource-rich interventions (for exam- 
ple, a learning specialist who works intensively with small groups of children) and with inter- 
ventions that require fewer resources (such as computer-based programs overseen by reading 
specialists or teachers on special assignment to supplement reading instruction). 

The cost per student for the average of the other supplemental reading services of- 
fered at the six sites was $1,780 and ranged from $1,050 to $4,890 per student. The range in 
the total resources provided by the supplemental services across the six sites depended on the 
type of supplemental services that were offered, which varied considerably in the resources 
they required. For example, computer-based programs serve groups of students on computers 
that the school also uses for other purposes, the costs of such programs tended to be relatively 
low (on average, around $940 per student in the study sites) when compared with more inten- 
sive programs. Reading specialists or other full-time school support staff who worked with 
small groups of students several times each week in a dedicated classroom tended to be more 
expensive (around $3,200 per student, on average, in the study sites). 

Table 5.3 also shows that although Reading Partners provides a rich set of resources to 
schools, the portion of costs borne by the school is substantially lower than for the other sup- 
plemental reading services provided. As already explained, the average cost to the school for 
Reading Partners was $710 ($480 to $1,270) per program group student, while the average cost 
of the other supplemental reading services borne by the school or school district was $1,700 
($1,040 to $4,890) per student. Most of the other supplemental reading services were funded 
solely by the school or school district. The few that were not fully funded by the schools relied 
on in-kind donations from volunteers or were subsidized by the program itself. 
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Reading Partners Evaluation 
Table 5.3 


Site-Level Costs of Reading Partners and Other Supplemental Services 


Site 3 

School Contribution per Student 13 ($) 

Total Resources per Student ($) 

Contribution Provided by School (%) 

Reading 

Partners 

Other 

Supplemental 

Services 3 

Reading 

Partners 

Other 

Supplemental 

Services 0 

Reading 

Partners 

Other 

Supplemental 

Services 0 

Site A 

690 

1,840 

3,450 

1,840 

20 

100 

Site B 

520 

1,850 

3,420 

2,230 

15 

83 

SiteC 

940 

2,680 

3,570 

2,680 

26 

100 

Site D 

1,270 

1,040 

5,190 

1,050 

25 

99 

Site E 

660 

1,310 

4,210 

1,980 

16 

66 

Site F 

480 

4,890 

2,740 

4,890 

17 

100 

Pooled 

710 

1,700 

3,610 

1,780 

20 

96 


SOURCES: MDRC calculations from cost data. 

NOTES: Costs are measured in 2012 U.S. dollars and are rounded to the nearest 10 in the tables and text. 

Averages have been weighted to account for the size of the program group (for Reading Partners) and students served at each site (for other 
supplemental services). 

All the Reading Partners costs in this table are those for the respondents in the program group and are adjusted for the larger dosage they 
received. The costs for supplemental services are those for all the students served at each site. 

a To prevent identification of individual schools or programs, site letter identifiers are not consistent across analyses. 
b Costs borne by school include the fee for services paid by schools. 

c Other supplemental reading services include small-group support from reading specialists, tutoring programs, homework help, and 
supplemental curricula. 



Thus, the portion of the costs borne by the school is greater for the other supplemental 
reading services than for Reading Partners, where most of the costs were covered by in-kind 
contributions from the volunteers and by funds from the Reading Partners organization itself. 
The portion of the costs borne by the school for Reading Partners was 20 percent on average (15 
percent to 26 percent), whereas the average portion of the costs of the other supplemental read- 
ing programs financed by the school was 96 percent (66 percent to 100 percent). Some of the 
other supplemental reading services offered at these schools may not be direct substitutes for 
Reading Partners because they serve students of differing levels of need, or a school may be 
required to provide a particular service (such as a Tier III reading specialist) where the principal 
is not able to select between the service and a program such as Reading Partners. 

Overall, schools provided students with a menu of supplemental reading services worth 
about $5,400 on average ($3,610 of Reading Partners services and $1,780 of other supplemental 
services). The average school was able to contribute around $2,400 worth of supplemental read- 
ing services per student for the students in their schools ($710 for Reading Partners plus $1,700 
in additional supplemental programs). The Reading Partners program, in part because it uses a 
volunteer-based model, provided schools with almost $3,000 worth of additional resources per 
program group student ($3,610 less the $710 contributed by the school) that the schools would 
otherwise not have been able to provide. 

Variability in Costs 

Table 5.3 also highlights the variation in the total resources devoted to Reading Partners 
across the six sites in the sample. The total resources used to provide Reading Partners ranged 
from a low of $2,740 to a high of $5,190 per program group student. The costs borne by the 
school ranged considerably, from a low of $480 per program group student to a high of $1,270 
per program group student. There were a number of sources of variation. First, there was varia- 
tion in the amount of time devoted by Reading Partners staff and AmeriCorps members, which 
was, in part, dependent on the number of students served in the school. Second, there was varia- 
bility in volunteer costs across schools, also depending on the number of students served, as 
well as differences in commuting time and transportation costs. At the school level, variation in 
costs depended on the amount of time school principals and teachers spent in coordination with 
the Reading Partners program, as well as some variation in space assigned to the program. The 
time devoted to referrals and formal or informal check-ins by the principal, for example, ranged 
from 30 minutes per year to over 10 hours per year across the sites. Teachers in all schools 
tended to spend some time on referrals or communicating with the site coordinator (or both). 
This time varied widely from 1 1 hours in total per year to over 3 hours per student per year. The 
space the schools were able to make available for the program also varied. As discussed in 
Chapter 2, most schools provided a dedicated classroom for the program, but some schools used 
space in the library or other available space rather than a standard classroom. 
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Variation in costs was also due to variation in the amount of services provided. Some 
sites offered more tutoring sessions per program group student than did others over the course 
of the school year. Sites that offered fewer sessions used fewer resources. The difference in the 
number of sessions provided per program group student between the sites with the highest cost 
and the lowest cost was about 10 sessions over the course of the year. 

However, for the most part the variability observed among schools was largely tied to 
scale — or the number of students served by the program. This is because serving more stu- 
dents allows the program to amortize its fixed costs (such as space and the site coordinator) over 
more students. The school sites with the lowest cost per student were those that served the high- 
est number of students and provided the highest number of sessions per student. 

Scale Test 

A sensitivity test was included to adjust for scale by using the operational maximum 
number of students based on the space and equipment recommended by Reading Partners for 
each reading center. The program recommends eight distinct spaces for tutor-student pairs to 
read and work together during each session. The reading centers tended to serve students for 
five class periods each day, four days a week. Accordingly, each site’s costs were adjusted to 
ascertain the costs if 80 students were served. This analysis is discussed in detail in Appendix D. 
The results, presented in Appendix Table E.l, show that the total cost per student drops to 
$2,390 on average ($2,230 to $2,700). Because the total costs borne by the school were as- 
sumed to be fixed (up to 80 students), the portion of the total resources used to implement Read- 
ing Partners borne by the school dropped from about 20 percent to 14 percent. This drop sug- 
gests that some schools could reduce cost per student substantially by serving more students. 

Thus, if a school has eligible students who are not served by the program, the site could 
improve the efficiency of the program by increasing the number of students served. However, 
the space available at a school may not be large enough to support 80 students or, alternatively, 
the community may not have sufficient volunteers to support expanded services. Additionally, it 
could be the case that a school would increase the time invested by staff when more students are 
involved, increasing the cost to the school. Yet, even a small increase in the number of students 
served may reduce variability in costs among sites and improve efficiency substantially. 


Conclusions 

Schools pay $320 per student out of pocket for Reading Partners and provide $390 of in-kind 
resources, such as space, to support the program. For this outlay, the school receives a program 
worth, on average, $3,610 per student ($2,740 to $5,190) to supplement reading instruction for 
students who need support. 
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The cost borne by the school to implement Reading Partners is about $1,000 lower per 
student than the average cost for other supplemental reading services being provided. The other 
supplemental reading services provided to the study sample varied in intensity and purpose, but 
all were financed almost solely by the schools or school districts. Reading Partners provided 
schools with a twice-weekly tutoring program for students who were half a year to two and a 
half years behind, in which most of the costs are borne by outside entities (that is, Reading Part- 
ners, AmeriCorps, and in particular, the volunteers). 

One caution should be noted. When comparing the cost numbers presented in this chap- 
ter with those in other reports, readers should make sure that the resources covered by the cost 
estimate are similar. In this cost analysis, all ingredients, including those contributed in-kind 
such as space, school staff time for collaboration, volunteer transportation, and volunteer time, 
were included in the total cost estimate. Many cost studies rely on budgetary data, ignoring in- 
kind resources, and thus might not be comparable with the numbers reported here. 

Overall, the findings of the cost study illustrate the high value of the Reading Partners 
program. The program used many resources (on average $3,610 per program group student), of 
which over half ($1,910) is contributed in-kind. Furthennore, the school bore only 20 percent of 
the costs of the program. Because Reading Partners relies largely on volunteers and is provided 
by an external organization, schools can capitalize on substantial resources without incurring a 
large burden. 
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Chapter 6 

Conclusions 


This evaluation finds a positive impact of the Reading Partners program on three different 
measures of reading proficiency, and the impacts appear to accrue to a wide range of students 
with differing incoming abilities, genders, and grades, including English language learners. 
Exploratory analyses also suggest that the impact of the program was largest for those students 
beginning the program with the lowest skills. Moreover, while the Reading Partners program is 
resource-intensive (as would be expected of a one-on-one tutoring program), a majority of the 
resources are in-kind contributions and schools and school staff bear a small portion of the 
financial burden of implementing the program. 

The findings from this evaluation are important because very few programs that have 
been rigorously evaluated at scale (using a randomized trial with a sample of more than 300 
students from multiple schools) have been shown to improve reading outcomes for elemen- 
tary school children. As shown in Table 6.1, the study team was able to identify only five 
programs that met these criteria, and they are divided into two groups — four one-on-one 
tutoring programs (Reading Partners, Experience Corps, Minnesota Reading Corps, and 
Reading Recovery) and one whole-school reform program (Success for All). The tutoring 
programs are similar in that they all provide one-on-one instruction and have a staff member 
who is fully committed to managing the tutoring program. 1 Success for All, on the other hand, 
includes a classroom-based curriculum, and coordinators must manage the many components 
of the program model. 2 However, the Reading Partners model is different, along a number of 
dimensions, from each of these. 

First, Reading Partners is unique among rigorously tested tutoring programs in that ser- 
vice delivery relies almost exclusively on community volunteers who devote relatively short 
amounts of time each week to tutoring. The Reading Recovery program uses highly trained, 
certified teachers to provide reading instruction to struggling elementary school students. Not 
surprisingly, the program has been very successful in improving the reading proficiency of very 
low-achieving first-graders. Minnesota Reading Corps relies on AmeriCorps members who 
commit to participate in the program either half time or full time and are provided with uniform 
and comprehensive training. 3 Similarly, Experience Corps volunteer members affiliated with the 


'Lee, Morrow-Howell, Jonson-Reid, and McCrary (2010); Markovitz, Hernandez, Hedberg, and Silber- 
glitt (2014); May etal. (2013). 

‘Quint et al. (2013); Quint et al. (2014). 

3 Markovitz, Hernandez, Hedberg, and Silberglitt (2014). 
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Reading Partners Evaluation 
Table 6.1 

Reading Programs with Rigorous Evidence Base 


Reading 

Program 

Program 

Description 

Population 

Served 

Service 

Providers 

Time 

Impacts for Students 
in Kindergarten or 
Grade 1 (Effect Size) 

Impacts for Students in 
Grade 2 or 3 (Effect Size) a 

ONE-ON-ONE TUTORING PROGRAMS 

Reading 

Partners 

Community 
volunteers work 
with students 
using a structured 
curriculum 

Elementary 
students 6-30 
months below 
grade level in 
reading 

Volunteers of 
varying 
backgrounds 
and ages 

45 minutes 
twice a 
week 


0.11- Sight-word efficiency 

0.09 - Fluency 

0. 10 - Comprehension 

Experience 

Corps 

Older adult 
volunteers tutor 
elementary school 
students 

Elementary 
students (grades 
1-3) at risk of 
academic failure 

Volunteers 
age 55+ 

60 to 160 
minutes per 
week 

0.10 - Decoding 
0.13 - Comprehension 

0.10 - Decoding 
0.13 - Comprehension 

Minnesota 

Reading 

Corps 

AmeriCorps 
members tutor 
elementary school 
students using 
scripted 
interventions 

Elementary 
students (K-3) at 
risk of academic 
failure 

AmeriCorps 

members 

20 minutes 
daily 

1 .06 - Letter sounds 
0.37 - Decoding 

0.10 - Fluency 

Reading 

Recovery 

Teachers provide 
supplemental, 
individualized 
instruction to 
students 

Lowest-achieving 
15%-20% of 
lst-grade readers 

Specially 
trained and 
supervised 
teachers 

30 minutes 
daily 

0.68 - Overall reading 
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Table 6.1 (continued) 


Reading 

Program 

Program 

Description 

Population 

Served 

Service 

Providers 

Time 

Impacts for Students 
in Kindergarten or 
Grade 1 (Effect Size) 

Impacts for Students in 
Grade 2 or 3 (Effect Size) 3 

WHOLE-SCHOOL REFORM PROGRAM 

Success 
for All b 

Classroom-based 
literacy instruction 
that includes 1:1 
tutoring in the 
context of a whole- 
school reform 
model 

All students 
in the school; 
lowest-achieving 
students in 
grades 1-3 who 
are eligible for 
tutoring 

Elementary 

school 

teachers 

1.5 hours 

instruction 

and 20 

minutes 

tutoring 

daily 

0.09 - Word identification 
0.18-0.35 - Decoding 

0.22 - Word identification 
0.33 - Decoding 
0.21 - Comprehension 


SOURCES: Borman et al. (2007); Lee, Morrow-Howell, Jonson-Reid, and McCrary (2011); Markovitz, Hernandez, Hedberg, and Bilberglitt 
(2014); May et al. (2013); Quint et al. (2013); Quint et al. (2014). 

NOTES: Programs in this table had to be serving at least 300 students across multiple schools and evaluated using a randomized controlled trial. 
Only impacts on standardized student assessments significant at p < 0.1 have been included. 

“The sample for the Reading Partners impacts provided in this column includes 2nd- through 5th-graders. The other evaluation samples in this 
column include 2nd- and 3rd-graders only. 

b Success For All findings come from three different reports (May et al., 2013; Quint et al., 2013; Quint et al., 2014). Findings for grades 2-5 
come from the randomized field trial, and those for kindergarten and grade 1 come from the ongoing i3 scale-up evaluation. 



AmeriCorps program commit varying amounts of time (between 4 and 15 hours a week de- 
pending on their location) to tutoring and receive between 15 and 32 hours of training. 4 

Reading Partners, on the other hand, empowers its school-based AmeriCorps members 
to run reading centers and manage and support a corps of adult volunteers, who, with minimal 
background or training, then provide services to students for up to a few hours each week. The 
reliance on community volunteers means that the Reading Partners program has the capacity to 
serve large numbers of students within a school (up to 80 in a given week) with the support of 
only one AmeriCorps member (the site coordinator), because several tutors can engage in tutor- 
ing at the same time. Despite the limited training and the relatively low level of commitment 
required of the Reading Partners volunteers, this study still found that the program had a posi- 
tive impact on students’ reading proficiency scores. 

Second, Reading Partners serves a wider range of students, particularly with respect to 
age and ability, than do the other tutoring programs that have been rigorously evaluated. Most 
elementary school reading interventions that involve tutoring focus almost exclusively on stu- 
dents in kindergarten through grade 3, and tend to find large impacts in kindergarten and 1st 
grade. Fewer have demonstrated impacts on students in upper elementary school grades (4 and 
5). Reading Partners, however, serves students through grade 5, and this evaluation finds a posi- 
tive impact of the program on the reading comprehension skills of students in the upper elemen- 
tary grades. Similarly, some programs, like Reading Recovery, target only the very lowest- 
achieving students. Reading Partners, on the other hand, serves students across a wider range of 
incoming reading abilities — six months to two and a half years below grade level. This means 
that students who are struggling but might not otherwise be served by the school are able to 
receive additional support. As shown in Chapter 4, Reading Partners is broadly effective across 
the various student subgroups explored by the research team. Impacts did not differ significantly 
between the lower elementary cohort (students in grades 2 and 3) and students in the upper ele- 
mentary cohort (students in grades 4 and 5), and although the impacts were largest for those 
students who were lowest-achieving when the study began, the impacts did not differ signifi- 
cantly between the various achievement quartiles. Finally, the program was also effective with 
English language learners, a group who face additional challenges in learning to read English. 

The low cost to schools of Reading Partners, especially relative to the costs of other 
types of supplemental reading programs, makes these findings particularly relevant for school 
administrators who are looking for practical ways to help the large numbers of elementary 
school students who are reading below grade level. For example, reports on Reading Recovery 
indicate that the total resource value of that program is about $4,400 to $7,300 per student, 


4 AARP Experience Corps (2014); Lee, Morrow-Howell, Jonson-Reid, and McCrary (2010). 
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depending on the implementation. 5 Similarly, the total resource value of Success for All is esti- 
mated to be around $13,000 per student. 6 Almost all the costs of these programs are borne by 
the schools. 7 Thus, volunteer tutoring programs like Reading Partners are able to channel re- 
sources to schools that otherwise might not have been available. 

There have been no cost studies of the Minnesota Reading Corps or Experience Corps 
programs, although they rely on a similar cost structure to that of Reading Partners. For exam- 
ple, the Minnesota Reading Corps program does not require schools to pay a fee for service and 
most of the resources are provided by the Reading Corps program, but schools are required to 
devote a percentage of time for a staff person who serves as the “internal coach” for the pro- 
gram. The internal coach is based at the school and provides ongoing literacy support and over- 
sight to the AmeriCorps tutors. 8 


Factors Contributing to the Observed Outcomes 

Given the findings described above, what factors in particular helped contribute to the pro- 
gram’s success? Previous research has indicated that a number of different program character- 
istics may lead to the success of tutoring programs in general and volunteer programs in par- 
ticular. Although the findings are somewhat mixed, six factors have been cited in the literature 
as instrumental to the success of volunteer tutoring programs: a structured curriculum, the 
alignment between the classroom and tutoring curricula, tutor background, tutor training, the 
consistency of the tutoring and the tutoring relationship, and one-on-one instruction. 9 Reading 
Partners has some, but not all, of these components. 

A Structured Curriculum 

Previous research has found that programs with either a structured curriculum or a 
structured division of time and activities had a larger impact on global reading outcomes than 
unstructured tutoring programs. 10 Other studies also note the importance of having structure in 
volunteer programs. 1 1 The structured use of time and materials is helpful because it ensures that 
consistent and unifonn instruction is given on a regular basis and because it allows tutors with 
limited backgrounds or training to implement the program successfully. Reading Partners has a 


5 Costs are adjusted to 2012 dollars. Hollands et al. (2013); Simon (2011). 

6 Costs are adjusted to 2012 dollars. Simon (2011). 

7 Simon (2011). 

Minnesota Reading Corps (2014). 

9 Elbaum, Vaughn, Hughes, and Moody (2000); Ritter, Barnett, Denny, and Albin (2009); Slavin, Lake, 
Davis, and Madden (2011); Wasik(1998). 

10 Ritter, Barnett, Denny, and Albin (2009). 

1 1 Slavin. Lake, Davis, and Madden (2011); Wasik(1998). 
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very structured curriculum covering a wide range of reading skills — and each skill is broken 
into a specific lesson with accompanying text, materials, and activities that are specific to that 
skill. Tutors widely said that the structure of the lessons allowed them to jump right in, even 
with minimal training. On the other hand, the Experience Corps program, which has a some- 
what similar model to Reading Partners and found similar impacts, includes no curriculum at all 
— volunteers use whichever curriculum is in place at the school where they are tutoring — and 
that program has been shown to be effective as well . 12 It is possible that this model works for 
Experience Corps because its members receive more training and spend more time tutoring than 
do Reading Partners’ volunteers. 

Alignment with Classroom Reading Curricula 

Some have argued that coordination between the curriculum used in the classroom and 
that used in the tutoring intervention is important because it provides children with continuity in 
their approach to learning and prevents any confusion or discrepant approaches to instruction . 13 
For example, the Experience Corps program closely integrates tutoring with classroom instruc- 
tion. 14 Reading Partners, on the other hand, does not. The Reading Partners curriculum in use at 
the time of this study was aligned with California state standards (and Reading Partners is work- 
ing to ensure that the program is well aligned with Common Core State Standards), but it is not 
tailored to the specific approaches or curricula used by individual schools. Instead, Reading 
Partners encourages staff members to communicate regularly with teachers and schools about 
individual student progress. However, the program managers and site coordinators who were 
interviewed for this study indicated that there was some variation in the types of communication 
that occurs and the coordination that results when knowledge is shared. In many cases, Reading 
Partners staff members merely provide progress updates to teachers or discuss troubling behav- 
ior; in other cases, the site coordinator might work with tutors in an attempt to align the stu- 
dents’ Reading Partners lessons with content being covered in the classroom. Thus, coordina- 
tion with the school curriculum does not appear to be a critical factor leading to the effects 
found in this evaluation. 

Selection and Background of Tutors 

Previous research suggests that tutor background and experience may have an impact 
on effectiveness. For example, one meta-analysis found that programs using college-going tu- 
tors produced especially large gains for students . 15 The tutors involved in Reading Partners have 


12 Lee, Morrow-Howell, Jonson-Reid, and McCrary (2010). 
13 Wasik(1998). 

14 Lee, Morrow-Howell, Jonson-Reid, and McCrary (2010). 
15 Elbaum, Vaughn, Hughes, and Moody (2000). 
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a wide range of backgrounds and experience, especially with regard to the amount of prior ex- 
posure they had to formal tutoring or working with struggling readers. However, the study team 
found no association between student test score gains and the tutor’s background. Thus, tutor 
background does not appear to be a key factor driving the results of this evaluation. 

Tutor Training 

Literature on volunteer tutoring indicates that tutor training is key to the success of a 
tutoring program, and training for the direct service providers is extensive in the Experience 
Corps, Minnesota Reading Corps, and Reading Recovery programs. 16 Experience Corps tutors 
receive between 15 and 32 hours of training; Minnesota Reading Corps members receive four 
days of training; and Reading Recovery teachers complete university-level coursework as part 
of their training. 17 While the training for Reading Partners full-time staff is comprehensive and 
ongoing, the amount of initial training that volunteers receive is minimal. As described in Chap- 
ter 4, most tutors receive only an orientation to the reading center and participate in a single 
shadow session before beginning to tutor students. The total time required for this up-front 
training is about an hour. Reading Partners does provide regular, ongoing training opportunities 
for tutors focused on particular challenges and student skills, but across the board Reading Part- 
ners staff reported that attendance at these training sessions was low. It seems unlikely that this 
single hour, during which the volunteer is not actively engaging with the curriculum or practic- 
ing techniques, is what is allowing these tutors to successfully help their tutees. On the other 
hand, site coordinators consider it one of their primary responsibilities to monitor and support 
tutors as necessary, and many tutors described their site coordinators as being particularly help- 
ful and responsive to issues that they encountered with students. A meta-analysis of mentoring 
literature found that ongoing support was associated with bigger impacts than initial training. 18 
Perhaps, while formal training is not the key, structured and accessible support infrastructures 
can partially explain how tutors with minimal training are successful, at least within the context 
of the Reading Partners program. 

Consistency of the Tutor-Student Relationship 

Research also suggests that the consistency of the tutor’s relationship with the student 
may be an important factor in improving academic outcomes. 19 Wasik cites the importance of 
this relationship as creating trust in the student, an understanding of student needs for the tutor, 


16 Elbaum, Vaughn, Hughes, and Moody (2000); Wasik (1998). 

17 Lee, Morrow-Howell, Jonson-Reid, and McCrary (2010); Markovitz, Hernandez, Hedberg, and Silber- 
glitt (2014); May etal. (2013). 

ls DuBois, Holloway, Valentine, and Cooper (2002). 

19 Elbaum, Vaughn, Hughes, and Moody (2000). 
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and a sense of stability to both. 20 Research on mentoring also found that a mentoring pair with a 
strong relationship led to improved academic outcomes, whereas mentoring absent a close rela- 
tionship did not significantly affect academic performance. 21 A snapshot of tutor consistency 
over a two-week period as pulled from a random sample of Reading Partners student folders 
indicates that individual tutor consistency was not particularly high in this study. Among stu- 
dents who were tutored four times or more over a two-week period, almost 60 percent saw more 
than two tutors, suggesting that tutor consistency was somewhat limited. Moreover, at the 
school level, the average session-to-tutor ratio varied from 1.25 sessions per tutor to almost 2.25 
sessions. The average number of assigned tutors and the average number of tutors seen over a 
two-week period was not related to outcomes at the site level. 

One-on-One Instruction 

Even if Reading Partners students are not consistently tutored by the same volunteer, 
they still receive consistent individualized tutoring. On average, students participated in about 
1.5 sessions per week, and all tutoring sessions were individual. Research on mentoring sug- 
gests that individualized attention can be key to helping students succeed in that it allows for 
relationship development. 22 As it relates specifically to reading instruction, evidence suggests 
that one-on-one tutoring is more effective than small-group instruction. 23 To that end, all of the 
tutoring-specific programs included in Table 6.1 involve one-on-one work with students. The 
one-on-one nature of these programs is likely part of the key to their success, but what makes 
Reading Partners unique is that this model is implemented with volunteers who devote relative- 
ly short amounts of time to tutoring. Not only does the program staff have to recruit and retain a 
large cohort of volunteers for each site, they also must have in place the infrastructure to ensure 
that one-on-one tutoring can happen during anticipated or unexpected tutor absences. Their abil- 
ity to consistently and reliably provide individualized tutoring in spite of volunteer turnover en- 
sures that Reading Partners services are delivered with fidelity to their model and likely helps 
explain some of the impacts found in this evaluation. The use of volunteers to provide one-on- 
one tutoring and the resulting minimal cost to schools also makes Reading Partners financially 
viable in communities with limited resources for supplemental programs. 

In summary, several of the characteristics that are cited as contributors to the success of 
tutoring programs are not a part of the Reading Partners model. The program does not require 
any prior tutoring or teaching experience, and volunteers come from a variety of backgrounds 
and experiences. Tutors participate in only limited formal training and tutor consistency is vari- 


20 Wasik(1998). 

2 'Bayer, Grossman, and DuBois (2013). 
22 Bayer, Grossman, and DuBois (2013). 
23 Slavin, Lake, Davis, and Madden (2011). 



able. Yet, the Reading Partners model appears to work because there are structures in place to 
help ensure that the primary goal of the program — to provide consistent, individualized tutor- 
ing to students — is not compromised by tutor characteristics or inconsistency and because the 
structured curriculum allows tutors to begin without much preparation. Site coordinators pro- 
vide ongoing support and guidance to tutors in lieu of formal training. Communication struc- 
tures are in place to help ensure that student learning is not interrupted simply because a differ- 
ent individual is conducting the tutoring, and backup systems are in place so that if a tutor is 
absent the session can often still be held, or if a student is absent the session can be rescheduled. 
While Reading Partners increases students’ reading skills by about a month and a half to two 
months using its current structure, it is possible that incorporating some of the characteristics of 
other tutoring programs that research has shown to be effective could result in larger impacts. 
For example, increasing the amount of formal training provided to volunteer tutors or increasing 
the amount of communication and coordination between Reading Partners and school staff 
might translate into a stronger tutoring program for students. 


Study Limitations 

While this evaluation was a rigorous one, there are some limitations to the conclusions that can 
be drawn from the findings and some questions that remain about the quality and importance 
of particular components of the Reading Partners program. First of all, the implementation data 
that were collected during the study emphasized a structural understanding of fidelity and how 
closely Reading Partners sites adhered to the national model. Tutors were observed to gain a 
more thorough understanding of the structural components of a tutoring session. While the 
research team noticed differences in how tutors explained concepts and modeled reading prac- 
tices, no attempt was made to evaluate the quality of particular tutoring approaches or styles. 
Therefore, the question remains as to whether higher-quality tutors and tutoring would result in 
larger impacts and what changes would need to be implemented in order to see higher-quality 
tutoring. Similarly, assessing the quality of the Reading Partners curriculum was beyond the 
scope of this project. Aside from knowing that the curriculum is highly structured and (accord- 
ing to Reading Partners) increasingly aligned with the Common Core State Standards, the 
evaluation did not assess whether the individual lessons are effective at teaching students spe- 
cific literacy skills and concepts. The only conclusion to be drawn is that it is designed in a 
way that allows tutors with minimal experience or training to successfully complete progres- 
sive sessions. 

Because data on program activities were collected directly from Reading Partners’ 
management information system, the study team was limited to the types of data that Reading 
Partners had already collected. One key question of interest is the role of tutor consistency in 
improving reading achievement, but the exploratory analyses conducted in this evaluation were 
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only able to use a school-level snapshot of tutor consistency over a two-week period. More 
detailed data on the number of different tutors seen over the course of the year or infonnation on 
the frequency of tutor absences is key to exploring the relationship between tutor consistency 
and program impacts. In particular, given the study team’s hypothesis that the one-on-one tutor- 
ing is the driving force behind the impacts found in this study, understanding whether simple 
one-on-one attention is sufficient or whether that attention needs to consistently come from one 
(or two) specific individuals would allow Reading Partners staff and other, similar practitioners 
to better serve students. 

It is also possible that the evaluation affected the implementation of Reading Partners. 
First, Reading Partners usually enrolls students on an ongoing, rolling basis as they are referred 
by teachers or other school staff, who prioritize the students who they think most need Reading 
Partners services. As a result of the study, however, schools identified and referred the whole 
pool of eligible students at the beginning of the school year; these students were then randomly 
assigned into the program or control group, and program group students were matched with 
tutors as quickly as possible. Absent random assigmnent, Reading Partners usually has a very 
targeted approach to identifying students for its program, a process that was disrupted by the 
needs of the study’s randomization and enrollment processes. At least one Reading Partners 
staff member indicated that some of the program group students were somewhat higher- 
achieving students than usually participate in Reading Partners at the study schools, likely a 
result of this change in procedure. If the observations of Reading Partners staff were true, given 
that exploratory analyses indicated the lowest-achieving students experienced the largest im- 
pacts, it is possible that impacts on the typical Reading Partners student would be larger than 
those found for the overall sample. 

Another way in which the evaluation could have affected these findings is by changing 
the equilibrium of services being provided to struggling readers in the study schools. In the ab- 
sence of Reading Partners, which brings additional resources to the school in the fonn of volun- 
teer tutors, supplemental services would have had a different distribution across the student 
population. That is, teachers in the study may have been allocating more supplemental services 
(not including Reading Partners) to the control group students than they would have done nor- 
mally. Small-group intervention services or other one-on-one tutoring programs represent the 
most common types of supplemental service provided to students in the control group. These 
services are typically costly with regard to staff time and place a large burden on school 
resources. (The cost study indicates that, on average, they cost schools $3,200 per student.) On 
the other hand, the cost of Reading Partners to schools is relatively low ($710 per student), 
meaning that more students can be served for the same amount of money. Thus, it is possible 
that absent Reading Partners, the “as is” condition at study schools would not have been able to 
serve as many students, thus decreasing the overall number of students receiving supplemental 
instruction at those schools. 
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A final limitation of this study is that student results and achievement are only meas- 
ured at the end of one year of services. Reading skills are important for students of all ages and 
grade levels, so programs with long-lasting effects are poised to be more useful to schools that 
are trying to improve reading skills among their students. Follow-up studies on these students in 
subsequent grades would indicate whether Reading Partners is one such program. 


Recommendations for Future Research 

The discussion above suggests several areas for future research and consideration by the Read- 
ing Partners program. 

First, more infonnation regarding tutor consistency and absences would help the pro- 
gram providers evaluate whether or not the lack of consistency reported by site coordinators is 
impeding student progress. To the extent that it is problematic, the Reading Partners program 
could consider putting structures in place to help mitigate this problem. 

Second, more infonnation about the quality of the tutor-student relationship would help 
Reading Partners staff identify program strengths and weaknesses and might surface specific 
areas in which more training is wananted. Relatedly, research on factors that increase attend- 
ance at tutor training might help Reading Partners devise more creative ways to deliver training 
to their community volunteers. Gathering more infomiation about tutor background and its rela- 
tionship to student outcomes could also help Reading Partners target its recruitment activities 
more effectively. 

In addition, more infonnation about the quality and effectiveness of the curricular mate- 
rials themselves is warranted, since the curriculum may be related to observed outcomes. Final- 
ly, future studies should examine whether or not impacts are sustained beyond a single year. 


Conclusions 

The Reading Partners program successfully improved students’ reading comprehension, sight 
word efficiency, and fluency by about 0.10 standard deviations, or one and a half to two months 
of growth, over the course of the school year. While the program is relatively resource- 
intensive, a majority of the resources are in-kind contributions, and the portion of the resources 
financed by the schools is relatively low — only 20 percent of the total cost of the program. 
Moreover, among sites that were included in the cost study, the cost of Reading Partners borne 
by the schools was substantially less per student than the cost of other supplemental reading 
services offered to students. Thus, this study provides additional evidence that volunteer pro- 
grams can work and that one-on-one tutoring is effective in improving academic outcomes. The 
results also suggest that effective volunteer tutoring programs may be a cost-effective option for 
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underresourced schools, because they bring additional resources to the school through commu- 
nity volunteers. Reading Partners manages to be effective even in the absence of oft-cited key 
components to successful tutoring, including, in particular, extensive tutor training and tutor 
consistency. Further research is required to understand whether improving these components of 
the model would affect the magnitude of the impacts and whether the impacts of Reading Part- 
ners are sustained for more than one year. 
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Appendix A 

Implementation Study Methods 




The qualitative data analysis coding scheme that was developed for the evaluation of Reading 
Partners was designed to align with the overall framework of the implementation study. The 
fidelity index is described in detail in Table A.l. A team-based approach was used to code all 
qualitative data (interview transcripts, notes, and observation write-ups). Each team member 
was assigned a set of codes to apply across all the qualitative data. The codes covered the 
following broad topics: instruction and delivery, Reading Partners staff roles and responsibili- 
ties, staff and tutor training, data-driven instruction, dedicated school space, program eligibility, 
treatment contrast and crossover, implementation context, volunteer tutor background and 
recruitment, Reading Partner staff background, school/district environment and characteristics, 
impact of the study on the program, Reading Partners history, and other contextual factors. 

After the systematic coding process, all team members developed analysis memos for 
each code they were assigned, which provided a summary of emerging themes, a school-by- 
school analysis of data that would be applied to the fidelity index (where applicable), and 
overall reflections and findings gleaned from the systematic review of the data. All qualitative 
data were analyzed using web-based, qualitative data analysis software. All analysis memos 
were discussed by several members of the study team and were revised based on feedback and 
discussion. The specific data used for the fidelity index were reviewed by at least two team 
members and the study’s principal investigator. 
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Reading Partners Evaluation 
Appendix Table A.l 
Fidelity Index Codebook 


Components and Indicators 

Possible 

Scores 3 

Criteria 

Data Sources 

Regular, one-on-one tutoring 
administered by volunteers 

Average number of tutoring sessions 
each week 

0,1,2 

>1.7 sessions/week = 2 
1.5-1. 7 sessions/week = 1 
<1.5 sessions/week = 0 

MIS data 

One-on-one: students are seeing tutors 
on an individual basis 

0,1 

SC indicated that all tutoring was one-on-one = 1 
Otherwise = 0 

SC interview 

Dedicated school space and 
use of materials 

Required materials that were present in 
reading center during observation 

0,1 

All materials = 1 
Missing any = 0 

Materials checklist 

Dedicated vs. shared school space 

0,2 

“Space that is enclosed and unshared during RP time’’ = 2 
Otherwise = 0 

Site observations 

Data-driven instruction 

Whether or not Rigby is administered 
three times, at beginning, middle, and 
end of school year 

0,1 

> 90% of eligible students took Rigby on 3 occasions = 1 
< 90% of eligible students took Rigby on 3 occasions = 0 

MIS data 

Whether or not Rigby is used to inform, 
customize, and update student’s 
Individualized Reading Plan (IRP) 
(including student’s progress in the 
curriculum) 

0,2 

Reports that midyear Rigby was used to either create action plan or 
advance students, with the assumption that IRP was updated = 2 

No mention of how midyear Rigby was used, or of updating IRP = 0 

SC, PM interviews 


(continued) 






Appendix Table A.1 (continued) 


Components and Indicators 

Possible 

Scores 3 

Criteria 

Data Sources 

Data-driven instruction (continued) 

Whether or not student work is 
documented after each tutoring session 

0,0.5, 1 

All “yes” = 1 
1 “no” = 0.5 
More than 1 “no” = 0 

Folder review 

Whether or not student data were 
shared with school principal 

0,1 

PM indicated sharing data with PR = 1 
PM did not indicate sharing data with PR = 0 

PM interview 

Rigorous and ongoing training 

Reading Partners staff participated in 
initial training 

0,1 

PM and SC reported participating in initial training = 1 
Either or both PM, SC reported no initial training = 0 

PM, SC interviews 

Do tutors receive in-service training? 

0,1,2 

Ratio of reported trainings to tutors: Training > tutors = 2; 
Training = Tutors = 1; 

Training < Tutors = 0 

Tutor interview 

Reading Partners SCs participated in 
ongoing, biweekly; or monthly training 

0,1 

SC reported participating in ongoing training = 1 
SC didn’t participate = 0 

SC interview 

Tutors participating in shadow session 

0,1 

All tutors at the site reported participating in shadow session = 1 
Any tutor indicated no shadow session = 0 

Tutor interview 

Instructional supervision and support 

Presence of PM who oversees a specific 
area of schools and SCs 

0,1 

Present PM = 1 
Missing PM = 0 

PM interview 

Presence of full-time SC who oversees 
operations at each school 

0,1 

Present SC = 1 
Missing SC = 0 

SC interview 


(continued) 






Appendix Table A.1 (continued) 


oo 


Components and Indicators 

Possible 

Scores 3 Criteria 

Data Sources 

Instructional supervision 
and support (continued) 




Qualitative indicators of SC serving as 0,0.5, 1 
liaison between program and partner 
schools 


SC lists this as key part of job = 1 

SC talks regularly with tutors, but does not describe as 
key aspect of job = 0.5 


SC interview 


SC does not list this as key part of job and has 
infrequent contact with tutors = 0 


SC discusses key roles as providing support/coaching SC interview 

(as opposed to monitoring) tutors = 2 

SC talks about training and monitoring tutors, but not really coaching, 
or SC says it’s hard to coach because doing a lot of tutoring, or 
mentions coaching/ supporting tutors but at the end of list of 
responsibilities = 1 

No mention of coaching or monitoring = 0 

Qualitative indicators of PM providing 0,1,2 PM lists this as key part of job = 2 PM interview 

coaching and support to SC . . .. v . __ . A 

PM describes monitoring SCs but not supporting 

them in other ways = 1 

No mention of coaching or supporting = 0 


Qualitative indicators of SC providing 0, 1 ,2 
coaching and support to tutors and 
students 


NOTES: MIS is management information system. SC is site coordinator. PM is program manager. PR is principal. RP is Reading Partners. 

“indicators with a maximum possible score of 2 are those indicators deemed most important for a faithful implementation of Reading Partners. 




Appendix B 

Impact Study Methods and Teacher Survey 




Obtaining, Cleaning, and Merging Administrative Data Sets 

All students had parental consent to participate in the study and for their records data to be 
collected. Demographic data, student attendance data, and state test score data were obtained 
directly from the participating districts for individual students. Data on student participation 
rates and tutor-student matches were obtained directly from Reading Partners. Raw data with 
student identifiers were kept on secure servers, and only key project staff had access to these 
data. In order to protect respondent confidentiality, student names and ID numbers were only 
stored together in one crosswalk file, which contained some or all of the following variables for 
each student: the student’s name, a district-assigned ID number, a Reading Partners-assigned ID 
number, an internal study-assigned ID number, and an encrypted unidentifiable ID number used 
for data processing and analysis. All data were cleaned in a multistep process that included 
(1) removing any student identifiers and replacing them with the encrypted ID, (2) identifying 
and resolving inconsistencies in each data source, (3) merging data across cleaned data sets, and 
(4) creating analytic variables to use in analyses. All data-cleaning programs were reviewed by 
at least two analysts. 


Standardization of Outcome Variables 

In order to compare students of different ages, grades, and states in the pooled impact analysis, 
steps were taken to standardize outcome variables. Documentation provided by the SAT- 10, 
TOWRE-2, and AIMSweb include various types of conversion tables from raw scores, as 
shown in Appendix Table B. 1 . 

For the SAT- 10 and TOWRE-2, the provided conversions were sufficient for analysis. 
However, the AIMSweb percentile conversions do not take into account the differing levels of 
difficulty across the different forms of the test. To standardize this outcome, a z-score was 
calculated for each student’s baseline and outcome scores using the baseline mean and standard 
deviation of scores for that grade-level and test form. 

District-provided state testing outcomes required additional standardization because 
scaled score ranges and proficiency level categories vary by state. Student proficiency scores 
were recoded into an indicator variable, representing whether or not students met the proficien- 
cy criteria established by their state. Scaled scores were converted to z-scores using a similar 
approach to that used with the AIMSweb scores. The 2013 control group mean and standard 
deviation for the student’s state and grade level were used for this standardization. 
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Appendix Table B.l 

Availability of Standardized Scores for Primary Outcomes 


Assessment 

Scaled 

Score 

Percentile 

Score 

Grade 

Equivalence 

Age 

Equivalence 

SAT- 10 

Available 

Available 

Available 

Available 

TOWRE-2 

Available 

Available 

Available 

Available 

AIMS web 


Available 




Imputation of Covariates and Baseline Reading Skills 

Consistent with the analysis plan, in the case of covariate missing values, imputation was 
conducted using the mean of the school and grade level to which the student belonged. The 
overall rate of missing data was quite small and thus multiple imputations were not used. If 
more than 5 percent of observations are missing data for a given covariate, the model includes a 
dummy to indicate whether a student’s data were imputed for this covariate. No outcome 
variables were imputed. 


Analytic Model 

For each outcome, the basic model used in the analysis to estimate impacts was as follows: 


where: 


Yjjk -y°Y-l,ijk + A ) Tijk + '^'YjY\kjBijk + 'Y j y2sXsijk + U k T i jk + £ijk( 1) 

k j S 


1 ‘J k = 1 if student i from grade j in school k is assigned to Reading Partners and 0 

otherwise 

Y j ,. k = the pretest score for student i from grade j in school k before random 
assignment 

Bijk = block dummy, 1 if student i from grade j is in a particular random assign- 
ment block, defined by grade block (2/3 or 4/5) and school k, and 0 
otherwise 

X sj k = the Sth other student-level covariates for student i from grade j in school k, 
these variables include the time lapse between baseline and follow-up 
testing as well as the demographic characteristics of age, gender, race, 
poverty status, special education status, and English language learner status 
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u Tij k = a random error term for the treatment group in school k, assumed to be 
independently and identically distributed; this term is included in the model 
to allow for variation in the impact of the program across schools 

Sijk = a student-level random error for student i from grade j in school k, assumed 
to be independently and identically distributed 


Correction for Multiple Hypotheses 

Multiple hypothesis testing was accounted for in the three primary outcomes of interest (reading 
comprehension, fluency, and sight word efficiency) using a procedure developed by Benjamini 
and Hochberg. 1 The adjustment is as follows: 


1 . Conduct N separate t-tests, each at the common significance level a. 

2. Order the p-values of the N tests from smallest to largest, where p(l ) <=p(2) <=. . .<= 
p(N) are the ordered p-values. 

3. Define k as the maximum j for which p(j) <=j/N*a. 

4. Reject all null hypotheses H 0( j)j = 1, 2,...,k. If no such A: exists, then no hypotheses are 
rejected. 


'Benjamini and Hochberg (1995). 
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Appendix Figure B.l 
Example of Teacher Survey 



l. 


Reading Partners Teacher Survey 

Student Information Form 


Has this student been assigned to your classroom for the past four weeks? 

YES (continue to the /'"'s |-jq If this student has not been assigned to your classroom for the past 

' ' ' next question } ' ' 




four weeks please stop here and go to the next student. 


For questions 2 and 3, please think about the amount of reading instruction that this student received in your 
classroom . By reading instruction we mean decoding, comprehension, vocabulary, and fluency. Do not include 
instruction that took place outside of your classroom (i.e., after school or via pull out). 


2 . 


What was the total amount of time (in minutesl that 
this student received instruction in reading each 
week? 

Print the number of minutes using all three boxes. For 
example, write 015 if you taught for 15 minutes. 



Total minutes per week spent on 

1 1 


reading instruction in vour classroom. 

1 1 


Of that time, how much time in minutes did you or an 
aide provide one-on-one help to this student in 
reading? 

Print the number of minutes using all three boxes. For 
example, write 015 if you taught for 15 minutes. 

Total minutes per week the student 
receives one-on-one reading help in your 
classroom. 


4. 


School personnel have indicated that the following supplemental reading programs or activities are available outside 
of vour classroom (i.e. after school or via pull out) at your school. Please mark all programs this student participated 
in this academic year and indicate the amount of time that they participated outside of vour classroom. 


Total Number of 
Weeks Attended 


Average Minutes 
Per Week 


□ 

Alternate Program #1 



weeks 




min/wk 

□ 

Reading Partners 



weeks 




min/wk 

□ 

Alternate Program #2 



weeks. 




min/wk 

□ 

Alternate Program #3 



weeks 




min/wk 


□ 


Another reading program or activity ( please specify) 


weeks 


min/wk 


5. Please report on this student’s behavior over the 


past month. 

Always 

Usually 

Sometimes 

Never 

This student completed homework assignments 

o 

o 

o 

o 

This student completed in-class assignments 

o 

o 

o 

o 

This student was attentive in class 

o 

o 

o 

o 

This student was disruptive in class 

o 

o 

o 

o 


Compared to other students in your school, please 
rate this child’s current academic performance in: 

Far Above 
Average 

Above 

Average 

Average 

Below 

Average 

Far Below 
Average 

Reading 

o 

o 

o 

o 

o 

Math 

o 

o 

o 

o 

o 

Overall 

o 

o 

o 

o 

o 


Thank you for your help! Please place this form in the envelope provided and store it securely in your classroom. It will be collected by the research team. 
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Appendix C 

Tutor Background Characteristics and 
Additional Impact Findings 




Tutor Background Characteristics, by Site 

Appendix Table C.l shows the background characteristics of the tutors at each of the 19 schools 
included in the study. 


Impact of Reading Partners on State Assessment Scores 

An exploratory analysis also examined the impact of Reading Partners on state assessment 
scores. This analysis has a number of limitations. First, the schools in the evaluation were 
located in three different states, each of which gave different assessments, measuring somewhat 
different skills. To combine assessments across states, scaled scores were converted to z-scores 
using the 2012-2013 control group mean and standard deviation for the students’ states and 
grade levels. However, even with these standardization efforts, assessments from different states 
may not actually be comparable with one another, limiting the conclusions that can be drawn 
If om these analyses. 

Second, student proficiency scores were recoded into an indicator variable, represent- 
ing whether or not students met the proficiency criteria established by their states. However, 
each state had its own criteria for determining proficiency; thus, this metric may not be 
measuring the same variables across states. Furthermore, because students who were referred 
to Reading Partners were half a year to two and a half years below grade level, it is not likely 
that a large percentage of students would obtain proficiency after 28 weeks of participating in 
the intervention. 

The findings from these analyses are shown in Appendix Table C.2. The program had 
no statistically significant impacts on the standardized state assessment scores in English 
language arts (ELA) or math, although the impact on ELA was positive, with an effect size of 
0.06. There was also no impact on the percentage of students reaching proficiency in either 
ELA or math. 


Additional Analyses Exploring Variation in Outcomes 

One way to explore the relative contribution of tutor consistency to student outcomes is to 
examine the relationship between a student’s reading growth and the number of tutors assigned 
to that student over the course of the year. It is important to note that the number of assigned 
tutors per student is not a true measure of tutor consistency because it does not take into account 
whether or not the assigned tutor was actually present for a student’s Reading Partners sessions. 
Additionally, any associations are correlational only, and are not evidence of causality, since the 
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Appendix Table C.l 

Tutor Background Characteristics, by Site 


Site a 

High School 
Students (%) 

College or 
Graduate 
Students (%) 

Working 
Adults (%) 

Retired 
Adults (%) 

Other 15 (%) 

Missing (%) 

Site A 

13.21 

30.19 

39.62 

7.55 

9.43 

- 

Site B 

7.04 

66.20 

11.27 

2.82 

11.27 

1.41 

SiteC 

20.59 

50.00 

17.65 

- 

8.82 

2.94 

Site D 

23.33 

35.00 

13.33 

10.00 

15.00 

3.33 

Site E 

20.41 

32.65 

20.41 

10.20 

10.20 

6.12 

Site F 

31.25 

34.38 

21.88 

6.25 

- 

6.25 

SiteG 

7.14 

42.86 

21.43 

7.14 

14.29 

7.14 

SiteH 

14.29 

35.71 

21.43 

7.14 

14.29 

7.14 

Site 1 

- 

83.33 

- 

- 

8.33 

8.33 

Site J 

19.74 

26.32 

28.95 

2.63 

13.16 

9.21 

SiteK 

28.13 

20.31 

15.63 

15.63 

10.94 

9.38 

Site L 

- 

30.00 

22.50 

10.00 

27.50 

10.00 

Site M 

16.18 

47.06 

7.35 

8.82 

10.29 

10.29 

SiteN 

5.56 

70.37 

1.85 

3.70 

3.70 

14.81 

SiteO 

22.22 

23.61 

22.22 

11.11 

5.56 

15.28 

Site P 

12.50 

10.00 

22.50 

15.00 

22.50 

17.50 

Site Q 

28.00 

10.00 

10.00 

16.00 

10.00 

26.00 

Site R 

20.55 

26.03 

16.44 

6.85 

2.74 

27.40 

Site S 

15.38 

15.38 

30.77 

7.69 

- 

30.77 

All sites 

17.20 

35.63 

17.65 

8.10 

10.32 

11.10 


SOURCE: MDRC calculations from Reading Partners management information system data. 

NOTES: Rounding may cause discrepancies in sums. 

a To prevent identification of individual schools or programs, site letter identifiers are not consistent 
across analyses. 

Tutors characterized as "Other" include, but are not limited to, caregivers, nonworking individuals, 
and those who describe themselves as "transitioning." 
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Appendix Table C.2 


Impact of Reading Partners on State Achievement Testing Outcomes 


Outcome 

Sample 

Size 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Impact 
Effect Size 

P- Value 

State reading achievement 

scaled score 2 

1,079 

0.08 

0.02 

0.06 

0.05 

0.06 

0.234 

State math achievement 

scaled score 2 

1,078 

0.01 

0.00 

0.01 

0.06 

0.01 

0.907 

Likelihood of achieving 
proficiency on state reading 
achievement test (%) 

1,079 

15.38 

14.54 

0.85 

2.01 

0.02 

0.674 

Likelihood of achieving 
proficiency on state math 
achievement test (%) 

1,078 

28.99 

31.75 

-2.76 

2.94 

-0.06 

0.348 


SOURCES: MDRC calculations using study team-administered baseline SAT-10 scores; district- 
provided demographic and achievement data; Reading Partners management information system data. 

NOTES: Calculations for this table used data for all students in the respondent sample with state testing 
outcomes. 

The model allows estimates to vary randomly across schools and controls for the random assignment 
block, the student-level pretest measure, the time lapse between baseline and follow-up testing, and 
student-level demographic covariates. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

a To account for differences across states, scaled scores have been normed to have a mean of 0. A full 
description of the the standardization of state testing variables can be found in Appendix B. 


number of tutors assigned was not determined randomly and there is no control group with 
which to compare the findings (that is, x number of tutors assigned to a program group com- 
pared with x number of tutors assigned to a control group). Appendix Table C.3 presents 
findings from an analysis looking at the relationship between the number of tutors assigned and 
student reading proficiency growth. 

As discussed in Chapter 4, Appendix Table C.4 shows how impacts on the three 
measures of reading proficiency varied by the school-level consistency of the tutors, based on 
folder review data. No consistent patterns between tutor consistency and outcomes were found 
in this analysis. 
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Appendix Table C.3 

Students' Reading Growth Based on Their Number of Assigned Tutors 


Number of Tutors 
Assigned to Student 


Comprehension 

Sight Word Efficiency 

Fluency 

Sample Mean 
Size Growth 

Standard 

Deviation 

Mean 

Growth 

Standard 

Deviation 

Mean 

Growth 

Standard 

Deviation 

1 

143 

18.96 

30.87 

0.84 

8.63 

18.17 

19.44 

2 

161 

16.57 

32.83 

2.60 

8.22 

21.82 

21.17 

3 

167 

17.37 

27.29 

1.00 

8.38 

20.93 

19.54 

4 

82 

20.47 

30.43 

3.06 

8.92 

22.62 

20.72 

5 to 8 

34 

21.94 

19.62 

0.73 

6.35 

15.82 

30.44 


SOURCES: MDRC calculations using study team-administered baseline and follow-up SAT-10, 
TOWRE-2, and AIMSweb scores; district-provided demographic data; Reading Partners 
management information system data. 

NOTES: Calculations for this table used data for all students in the respective subgroups of the 
respondent sample. Sample sizes for individual outcomes may fall short of the reported sample 
sizes because of missing or unusable data. 

Mean growth reflects the average increase in scores from pretest to posttest. Analyses used 
scaled scores for the SAT-10 (comprehension) and TOWRE-2 (sight word efficiency) and raw 
scores for the AIMSweb (fluency). 


Finally, as discussed in Chapter 4, Appendix Table C.5 shows how growth in reading 
proficiency on the three outcome measures was related to the background characteristics of the 
student’s primary tutor. Again, no consistent patterns were found. 


Subgroup Analysis of Primary Impacts Based on Reading 
Partners Target Status 

While Reading Partners serves a wide variety of students, it identifies “target students” as those 
who are 6 to 30 months below grade level in reading (as measured by the Rigby PM Bench- 
mark assessment), who are at least conversationally fluent in English, and who do not have an 
Individual Education Program (IEP). At each of its reading centers, Reading Partners aims for 
target students to make up at least 80 percent of all students served, and in the sample for this 
study, approximately 75 percent of respondents met those criteria. One exploratory question of 
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Appendix Table C.4 

Program Impacts, by School Level of Tutor Consistency 


Comprehension Sight Word Efficiency Fluency 

Impact Impact Impact 

School Level of Sample Effect Effect Effect 


Tutor Consistency 11 

Size 

Size 

P-Value 

Size 

P-Value 

Size 

P-Value 

Low 

385 

0.03 

0.736 

0.12 * 

0.080 

0.14 ** 

0.048 

Moderate 

424 

0.03 

0.643 

0.08 

0.236 

-0.03 

0.692 

High 

357 

0.30 

** 0.017 

0.14 ** 

0.045 

0.15 * 

0.089 


SOURCES: MDRC calculations using study team-administered baseline and follow-up SAT- 10, 
TOWRE-2, and AIMSweb scores; district-provided demographic data; Reading Partners 
management information system data; review of Reading Partners folders. 

NOTES: Calculations for this table used data for all students in the respondent sample. Sample sizes 
for individual outcomes may fall short of the reported sample sizes because of missing or unusable 
data. 

The impact analyses for student reading achievement were conducted using scaled scores for the 
SAT-10 (comprehension) and the TOWRE-2 (sight word efficiency) and sample-normed scores for 
the AIMSweb (fluency). The model allows estimates to vary randomly across schools and controls 
for the random assignment block, the student-level pretest measure, the time lapse between baseline 
and follow-up testing, and student-level demographic covariates. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Differences between 
subgroups are not statistically significant. 

a Schools with low tutor consistency are those in the lowest tercile of average sessions per tutor 
from the review of Reading Partners folders; schools with moderate tutor consistency are those in the 
second tercile of average sessions per tutor; schools with high tutor consistency are those in the third 
tercile of average sessions per tutor. 


this evaluation is whether or not Reading Partners has a different impact on those students who 
are designated as target students. Appendix Table C.6 shows the results of this subgroup 
analysis: 

• For the SAT- 10 (which measures reading comprehension) and the AIMSweb 
(which measures oral fluency), the impacts for both target and nontarget 
students are similar in magnitude to the main impacts, though most are not 
statistically significant. 
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Appendix Table C.5 

Students' Reading Growth Based on Background Characteristics of 
Their Reading Partners Tutors 




Comprehension 

Sight Word Efficiency 

Fluency 

Background of 
Primary Tutor* 

Sample 

Size 

Mean 

Growth 

Standard 

Deviation 

Mean 

Growth 

Standard 

Deviation 

Mean Standard 
Growth Deviation 

High school student 

70 

16.36 

29.72 

3.37 

6.81 

18.54 

25.43 

College or 
graduate student 

207 

19.27 

30.48 

0.43 

8.80 

19.12 

19.51 

Working adult 

105 

19.81 

30.18 

2.07 

8.64 

20.12 

21.20 

Retired 

79 

19.16 

31.07 

2.81 

8.67 

24.49 

22.62 

Other* 3 

75 

15.17 

26.67 

1.60 

8.63 

22.63 

20.07 

Missing 

58 

17.81 

29.41 

1.68 

6.75 

18.83 

19.86 


SOURCES: MDRC calculations using study team-administered baseline and follow-up SAT-10, 
TOWRE-2, and AIMSweb scores; district-provided demographic data; Reading Partners 
management information system data. 

NOTES: Calculations for this table used data for all students in the respective subgroups of the 
respondent sample. Sample sizes for individual outcomes may fall short of the reported sample sizes 
because of missing or unusable data. 

Mean growth reflects the average increase in scores from pretest to posttest. Analyses used scaled 
scores for the SAT- 10 (comprehension) and TOWRE-2 (sight word efficiency) and raw scores for 
the AIMSweb (fluency). 

“Among students' assigned tutors, primary tutors are those who had the longest assigned 
relationship with the student. 

b Tutors characterized as "Other" include, but are not limited to, caregivers, nonworking 
individuals, and those who describe themselves as "transitioning." 


• For the TOWRE-2 (which measures sight word efficiency), the impact on 
target students is slightly smaller (though not statistically significant) than for 
the full sample, but the impacts for the nontarget students are slightly larger. 

• Differences between these subgroups are not statistically significant. 
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Appendix Table C.6 


Subgroup Analysis of Primary Impacts 
Based on Reading Partners Target Status 




Comprehension 

Sight Word Efficiency 

Fluency 

Reading Partners 
Target Status 

Sample 

Size 

Impact 

Effect 

Size P-Value 

Impact 

Effect 

Size 

P-Value 

Impact 

Effect 

Size P-Value 

Target student 

853 

0.09 0.102 

0.06 

0.136 

0.11 ** 0.020 

Not a target student 

313 

0.12 0.285 

0.26 *** 

0.003 

0.10 0.309 


SOURCES: MDRC calculations using study team-administered baseline and follow-up SAT-10, 
TOWRE-2, and AlMSweb scores; district-provided demographic data; Reading Partners 
management information system data. 

NOTES: Calculations for this table used data for all students in the respective subgroups of the 
respondent sample. Sample sizes for individual outcomes may fall short of the reported sample sizes 
because of missing or unusable data. 

The impact analyses for student reading achievement were conducted using scaled scores for the 
SAT-10 (comprehension) and the TOWRE-2 (sight word efficiency) and sample-normed scores for 
the AlMSweb (fluency). The model allows program impact estimates to vary randomly across 
schools and controls for the random assignment block, the student-level pretest measure, the time 
lapse between baseline and follow-up testing, and student-level demographic covariates. 

A two-tailed t-test is used for all statistical tests presented in this table. Statistical significance 
levels between the program and control groups are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Differences between subgroups are not statistically significant. 
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Appendix Figure C.l 
Student Reading Growth 




• Program Group •••«••• Control Group 

SOURCES: MDRC calculations using study team-administered baseline and follow-up SAT-10, TOWRE-2, and AIMSweb scores; district- 
provided demographic data; Reading Partners management information system data. 


NOTES: Growth analyses were conducted using scaled scores for the SAT-10 (comprehension) and raw scores for the TOWRE-2 (sight word 
efficiency) and AIMSweb (fluency). Since the equivalence of baseline test scores of the two groups has been established, a common pre-test 
score is assumed for both. 



Appendix D 

Cost Study Methods 




The cost study conducted for this evaluation used the ingredients method to calculate costs. The 
ingredients method consists of three main steps. First, all ingredients that are used in implement- 
ing a program are identified, described qualitatively, and specified by quantities. All ingredients 
are specified based on the requirements for replication. To replicate the particular implementa- 
tion that produced a specific result, persomiel and the amount of time they devoted to providing 
services must be identified, as well as training requirements, facilities that were used to provide 
services or training, materials used, and any transportation or food or other services that were 
provided. All ingredients are included regardless of who paid for or contributed them . 1 


Identification of Other Supplemental Reading Services 

At all six sites that were included in the cost study, schools had other supplemental services or 
programs (in addition to the Reading Partners program) to provide additional reading instruction 
beyond the regular classroom teaching. Services were provided to all students in need of 
additional support, irrespective of whether they had been included in the Reading Partners study 
or whether they were assigned to the study’s program group or the control group. These 
supplemental reading services varied from reading specialists to tutoring programs to home- 
work help. Some of the supplemental reading services offered at these schools may not be 
comparable with Reading Partners in tenns of providing policy alternatives, because they serve 
students of differing levels of need (for example, more than two and a half years below grade 
level) or they may not be optional and so do not lend themselves to cost-effectiveness compari- 
sons. Thus, the costs of other supplemental reading services are included here to provide a 
context for the cost of Reading Partners and the school environment in which the program is 
implemented. 


Ingredients Data Collection 

The cost study collected data on all ingredients that were used in the Reading Partners program, 
including the time contributed by volunteers to provide tutoring to students and to commute to 
the school. Before conducting interviews with Reading Partners staff, the research team collect- 
ed data from publicly available sources such as websites and published reports to prepare a list 
of potential ingredients at each school site. Data from the implementation study helped the 
research team identify the materials used at each site. Data from the teacher surveys were used 
to identify the other supplemental reading services provided at each school. Program managers 
from Reading Partners at each site were then contacted for interviews. The interviews were held 
by phone and focused mostly on the numbers and types of personnel used to implement the 


Levin and McEwan (2001). 
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program. The program managers were also asked to describe the other supplemental reading 
services and to provide contact information for an individual involved in providing the other 
services. Often, a reading coordinator, reading specialist, or the school principal was inter- 
viewed regarding the school’s other supplemental reading services. 

Each ingredient was described in tenns of the quantity and qualities of the ingredient 
needed to determine the value or price. For example, questions were included about the experi- 
ence, education, and responsibilities of the program managers and site coordinators and other 
Reading Partners staff who were involved in providing the program services at the school. 
Descriptions were requested regarding the program’s use of time from teachers and principals 
because the time invested by these school-based personnel may have been important to the 
success of the program. The roles, qualifications, and time devoted by the volunteers were 
included. Even though the tutors volunteer at no cost to the school, the volunteers are a neces- 
sary input for the program to work and their services have a value that is included in this 
analysis. In essence, the estimated cost of volunteer time is what they would need to be paid in 
the market if they were unavailable as volunteers. Descriptions of space used to provide the 
program and any materials used by the program were also collected. 


Pricing the Ingredients 

The second step in the ingredients method is to assign prices to each ingredient based on market 
prices (in this case, for 2012). National prices are often the best option to account for prices 
when the intervention is applied in different locations so that the results are not dependent upon 
local variation in prices among different locations. If market prices are not available for an 
ingredient, a “shadow” or underlying price is used from the most similar resource with a market 
price. For example, no market rental rate is available for classroom space because classrooms 
are not typically rented. Therefore, to approximate the annual rental value of the space, the 
construction cost plus the costs for furnishings and interiors are priced and amortized over the 
life span of the building. 

In the cost study, each program ingredient was matched with its national price or cost in 
2012 dollars. When necessary, prices that were available only for other years were adjusted to 
2012 dollars using the Consumer Price Index. Ingredients that are used for longer than one year 
(such as initial training of tutors, facilities, books, and computers) were accounted for by their 
annualized costs, amortized with an interest rate of 3 percent. Tutor time for the initial training 
and the shadow session was amortized by the site- level average tutor participation if longer than 
one year; the ongoing training that occurred on a regular basis was viewed as an annual cost. 

The estimated cost of the other supplemental reading services provided at each school 
followed the same pricing approach. Each ingredient for the supplemental services was matched 
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with a national price. In some cases, specific prices were obtained from the program developer, 
such as the price of the program license and the price of the training package. The total cost and 
the cost per student for each program or service identified were calculated at the six schools in 
the sample. Each site had between one and four other supplemental services included in the 
analysis, and in addition to the Reading Partners programs at each school, there are 12 supple- 
mental reading programs in total. The costs were aggregated at the site level rather than reported 
individually in order to protect school confidentiality. 

Personnel 

The Reading Partners site coordinator and outreach coordinator positions were staffed 
by AmeriCorps members. These individuals were young, recent college graduates with little to 
no additional experience. To price these positions, the national average earnings amount for 18- 
to 24-year-olds with a bachelor’s degree was obtained from the U.S. Census Bureau, Current 
Population Survey, 2012 Annual Social and Economic Supplement. Benefits were estimated at 
3 1 percent based on the Bureau of Labor Statistics. Because this price may be higher than what 
the individuals received from AmeriCorps for their work, a sensitivity test was included that 
used the annual salaries and compensation information from the Reading Partners hiring 
postings that were available online. The hiring posts were specific to particular locations. Thus, 
the salaries were adjusted to national prices using the index of Regional Price Parities (2006- 
2010) and then the salaries were averaged for a national average salary rate. 2 

Based on the national statistics reported by the Bureau of Labor Statistics, the Depart- 
ment of Labor, and the National Center for Education Statistics, the hourly pay rate of school 
staff was transformed by dividing their annual salary and benefits by the number of hours in an 
academic year (that is, 1 ,440 hours from 1 80 days per year and 8 hours per day). 

Volunteer tutor time was valued at the same rate as paraeducators or teacher’s aids. This 
price was selected because the duties of the paraeducator best matched those of the volunteer 
tutor. Because Reading Partners relies on volunteers to serve students, upper and lower bounds 
were estimated for the value of volunteer time. The lower bound was estimated using the 
federal minimum wage. This wage was selected because the program advertises that its success 
is not dependent on the qualifications of the tutor. If all tutors were high school students, this 
price would represent the opportunity cost of the time contributed to the program. However, the 
program relies on tutors from various backgrounds. 

The upper-bound estimate used the tutor background information provided in Chapter 2 
(Figure 2.3). The time of any volunteer who was not enrolled in high school, college, or 


2 Aten, Figueroa, and Martin (2012). 
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graduate school was valued as the average hourly wage of an individual with a college degree 
using the average of median hourly wages for a person with a bachelor’s degree across all age 
groups expressed in 2012 U.S. dollars. 3 The other tutors, who were students, were valued at the 
paraeducator wage (which is very similar to the hourly wage of a high school graduate). While 
this estimate captures some of the variation in tutor education levels, the findings from the 
impact analysis in Chapter 4 find no clear indication that the background of a student’s primary 
tutor has any effect on reading abilities. 

The cost of transportation was included using the federal reimbursement rate per mile 
for driving or the price for a round trip on public transportation. 4 The cost for public transporta- 
tion is assumed to be $5 per round trip. 

Facilities 

In tenns of school facilities, the construction cost of a new school building was adjusted 
to include the site, site development, furnishings, and interiors. The price per square foot was 
amortized over 30 years with an interest rate of 3 percent. This annual cost per square foot was 
further divided by the number of hours in an academic year (that is, 1 ,440 hours, or 1 80 days 
with 8 hours per day) to get the cost of school facilities per square foot per hour. The national 
median price for construction costs of elementary schools was collected from the 19th School 
Construction Report , 5 and the adjustment index is based on national statistics provided by the 
National Clearinghouse of Educational Facilities (20 12). 6 

Materials and Equipment 

The prices for all materials and equipment (for example, desks, chairs, computers, 
printers) were collected from publicly available sources and expressed in 2012 dollars. The 
Consumer Price Index was used as needed to adjust the prices to 2012 dollars. To reduce the 
burden of interviews, materials that varied in quantity with the number of students served — 
such as pens, Post-it ® notes, bookmarks, student boxes, student folders, and so forth — were 
priced at $50 per student. Books used in the Reading Partners reading center and distributed to 
students were priced at the national average price of a children’s book used in public schools 
in 2012. 7 


3 U.S. Census Bureau (2012). 

4 Data on the federal reimbursement rate per mile came from the Internal Revenue Service (2011). 

5 Abramson (2014). 

6 See www.efc.gwu.edu. 

7 SLJ (2013). 
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As described in Chapter 2, an important component of Reading Partners is that data in- 
form the services provided to students, which is known as being “data-driven” in practice. The 
site-level ingredients included the Rigby PM Benchmark test kit, which is used three times per 
year to assess students’ reading levels and progress. The price of the Rigby PM Benchmark test 
kit in 2014 was transferred to 2012 dollars. During the interviews, the program managers 
reported that the Rigby kit was used for more than three years. The cost study assumes that the 
kit would require replacement after five years. The cost of the kit was amortized for five years 
using a discount rate of 3 percent. 


Calculating Costs 

The third step in applying the ingredients method is to estimate the total cost of the program and 
the cost per student. These costs can then be distributed across various entities or sponsors 
according to how they were financed. If an analysis includes cost-effectiveness, the fourth step 
is to calculate the cost-effectiveness ratios and examine the results comparatively. The cost 
study focused on estimating the cost of Reading Partners and the cost of the other supplemental 
reading services from the perspective of the school sites. A comparative cost-effectiveness 
analysis is not included because the study did not focus on the cost-effectiveness of other 
reading programs. 

Categories of cost estimation in this analysis include the value of total resources for all 
students served, cost per session, cost per program group student, and the total cost for the 
program group for each site. 


Limitations of the Cost Analysis 

As described above, the costs are sensitive to the valuation of volunteer time. Some limitations 
to the analyses should be noted. The main analysis used the same price ($15.14 per hour) for 
tutor time based on the national average wage of a paraeducator. By using the same price for all 
tutors, it could be possible that the true cost to replicate each site is not accurately estimated 
because the variability in tutors’ qualities, qualifications, experience, reason for tutoring, and so 
forth is not reflected. These qualitative characteristics of the tutors may be important for 
replication. However, as stated above, exploratory analysis did not find that tutor background 
was related to the students’ reading growth. It may be valuable in future analyses to collect 
more data on the volunteer tutors. 

Some sites used Federal Work-Study students from local colleges, in addition to com- 
munity volunteers, to provide tutoring. The time spent tutoring by Federal Work-Study students 
was not easily distinguished from the time contributed by volunteers. Thus, the distribution 
analysis treated all tutors as volunteers. If data were available on the number of sessions 
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provided by each type of tutor, the distribution of the financial burden of the program costs 
could reflect the portion borne by the U.S. Department of Education. Additionally, the portion 
of the costs borne by the volunteers would drop slightly. 

The upper-bound estimate of the volunteer time used the average wage of an individual 
with a bachelor’s degree for all nonstudents and the average paraeducator wage for all student 
tutors. This analysis could be improved by using the national average wages for high school 
graduates, college enrollees, college graduates, and graduates with advanced degrees. These 
prices were not used because the data were not detailed enough to identify precisely which 
degree each tutor had obtained. Rather than introducing additional assumptions into the analy- 
sis, the paraeducator wage was used for all student tutors. 

The cost study prices the time contributed by teachers and principals as the national av- 
erage wages earned by all individuals in those positions. Given the minimal involvement of 
these individuals, the prices used were not differentiated to reflect differences in qualifications 
and experience of the teachers and principals at each school site. Additionally, the cost study 
sought to minimize data collection to reduce the burden of interviews. As a result, the cost study 
did not inquire about the qualifications and experience of each teacher and the principals in the 
six school sites in the sample. 

Another potential source of data for the cost study was a biannual celebration of stu- 
dents’ progress in each site. However, the cost study could not fully investigate these two 
celebrations to identify who attended them. The celebrations were not deemed central to the 
successful implementation of the program and, therefore, in order to minimize the burden of 
data collection, the time spent by volunteers and possibly parents to attend the events was not 
included. 

The cost study team was able to collect observational data, which were very informa- 
tive, for one site in the sample. However, this site visit occurred after many of the site inter- 
views had been completed. Thus, some ingredients that may have been included in the imple- 
mentation were not included in the cost study. For example, one site visit revealed that the 
reading center had a comer devoted to a book drive, from which students could take books 
home to build their own libraries. When the student returned a questionnaire about the book, the 
student received a star on a chart and another book. The books were donated to the program 
from community members and organizations. The cost study was not able to collect data on this 
aspect of the program. 
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Scale Test 

Many of the ingredients that were used to implement the Reading Partners program were 
needed in order to provide the program services, even if there were fewer students than could be 
served. Thus, the cost per student for each site is closely tied to the number of students served, 
which then causes variability in the cost per student. Based on the space required for the reading 
center and the program’s recommended furnishing of the space, the program is designed to 
serve a maximum of 80 students per reading center (eight students over five periods per day for 
two days each per week). The total number of students served at each site, including those who 
were not in the program group (for example, kindergarteners, first-graders, and students who 
enrolled at the school after study recruitment had been completed), varied from approximately 
40 to 80 students. To estimate the lower bound of the cost per student at each site, an analysis 
was included in which each site’s cost per student was calculated as though the site operated at 
full capacity serving 80 students. 

Additional assumptions that were required for the analysis include: 

• The workloads of the site coordinator, program manager, and regional execu- 
tive director at each site do not change. 

• The maximum capacity of an outreach coordinator is assumed to be 600 
students. 

• The number of sessions per student received at each site is constant. 

• The volunteer body remains the same, but each tutor volunteers for addition- 
al sessions to meet the increased demand. 

• The working time of the school staff (teachers, principals, vice principals, 
and reading coordinator) remains constant. 

• The space required, which is designed to serve 80 students, remains constant. 
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Appendix E 

Additional Cost Findings 




Comparing Findings with Reading Partners Costs at Scale 


To investigate the lower bound of the cost per program student at each site, additional analyses 
were conducted as though the site operated at a full capacity of 80 students. 1 As shown in 
Appendix Table E.l, under all of these assumptions the average cost per student drops consid- 
erably, from $3,610 to $2,390, with a range of $2,230 to $2,700 per student per site. These 
results equate to about a 34 percent reduction in costs, on average. As a school expands its scale 
to approach the maximum capacity, 


the fixed costs for the school, such as 
facilities and working time of Reading 
Partners staff and school staff, are 


Reading Partners Evaluation 
Appendix Table E.l 


shared by more students, leading to an 
increase in efficiency. However, it is 
unknown whether the expansion 
would affect the impact of the 
program. The total cost borne by the 
school is unchanged based on the 
assumptions; however, the portion of 
the total cost borne by the school 
drops dramatically as the increasing 
part of total cost is mainly borne by 
the volunteers and Reading Partners 
itself. 

This sensitivity test illustrates 


Reading Partners Cost per Student: 
Main Study Findings and at Scale 


Cost per Student ($) 


Site a 

Main Findings 

At Scale b 

Site A 

3,450 

2,230 

Site B 

3,420 

2,550 

SiteC 

3,570 

2,350 

Site D 

5,190 

2,240 

Site E 

4,210 

2,700 

Site F 

2,740 

2,270 

Pooled 

3,610 

2,390 


the variability in resource use among SOURCES: MDRC calculations from cost data. 


sites regardless of scale. While scale 
and resource use are likely related, 
this test shows how much of the site- 
level variability is driven by scale 
rather than by implementation dif- 
ferences in resource allocation, such 
as volunteer travel time, principal or 
school staff time, and the size of the 
facilities devoted to the reading 
center. It is possible, though, that an 
increase in the number of students 


NOTES: Costs are measured in 2012 U.S. dollars and are 
rounded to the nearest 10 in the chapter tables and text. 

Averages have been weighted to account for the size 
of the program group at each site. 

All costs are those for the respondents in the program 
group and are adjusted for the larger dosage they 
received. 

a To prevent identification of individual schools or 
programs, site letter identifiers are not consistent across 
analyses. 

b "At scale" assumes Reading Partners serves 80 
students per site. Additional assumptions are described in 
Appendix D. 


'A full explanation of the assumptions used to estimate costs at scale appears in Appendix D. 
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would require an increase in school staff time to identify students and to collaborate with the 
program staff. 


Sensitivity of Reading Partners Cost to Changes in Valuations of 
Volunteer and AmeriCorps Members’ Time 

The research team also explored the sensitivity of the cost findings to differences in the value of 
personnel based on different likely scenarios. 2 The results of this sensitivity test are provided in 
Appendix Table E.2. The average cost per program group student, as reported in Chapter 5, is 
$3,610 (with a range of $2,740 to $5,190). When volunteer time is valued at the rate of the 
federal minimum wage, a lower-bound estimate compared with the main analysis, the cost per 
student drops by about $600. The upper bound used a higher rate for volunteer time based on 
the assumption that all volunteers who were not students had bachelor’s degrees. This high 
estimate is about $300 higher than the main findings. 

An additional sensitivity test was included to test for uncertainty around the price used 
to value the AmeriCorps members who serve as Reading Partners site coordinators and out- 
reach coordinators. The main analysis used the average earnings of recent college graduates as 
their annual salary. However, the actual salary and tuition benefits provided may have been 
lower than the national average wages. Thus, the sensitivity test used wage and tuition infor- 
mation that was available from the Reading Partners website to obtain an average salary. When 
this price is used, the cost per student drops by about $240, from $3,610 to $3,370. 

The main analyses use prices that reflect the broader marketplace and are likely closer 
to replication costs than the lower AmeriCorps member wage or minimum wage for the 
volunteers. These sensitivity analyses provide an estimate of how the range of the costs might 
vary. Because these tests focused on the uncertainty of the value of volunteers’ and AmeriCorps 
members’ time, the costs borne by the school were not changed. 


‘A full description of the sensitivity check appears in Appendix D. 
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Reading Partners Evaluation 
Appendix Table E.2 

Sensitivity Tests of Staff Prices for AmeriCorps 
Members and Volunteers 


Site" 


Cost per Student ($) 


Main Findings 

AmeriCorps b 

Volunteer Low c 

Volunteer High d 

Site A 

3,450 

3,210 

2,900 

3,740 

Site B 

3,420 

3,220 

2,790 

3,640 

SiteC 

3,570 

3,350 

2,950 

3,670 

Site D 

5,190 

4,840 

4,550 

5,690 

Site E 

4,210 

3,960 

3,580 

4,710 

Site F 

2,740 

2,570 

2,210 

3,150 

Pooled 

3,610 

3,370 

3,010 

3,930 


SOURCES: MDRC calculations from cost data. 

NOTES: Costs are measured in 2012 U.S. dollars and are rounded to the nearest 10 in the 
chapter tables and text. 

Averages have been weighted to account for the size of the program group at each 
site. 

All costs are those for the respondents in the program group and are adjusted for the 
larger dosage they received. 

a To prevent identification of individual schools or programs, site letter identifiers are 
not consistent across analyses. 

b The AmeriCorps sensitivity test uses the AmeriCorps stipend as the price of the site 
coordinators and outreach coordinator positions. 

c The "Volunteer Low" sensitivity uses minimum wage as the price of volunteer time. 

d The "Volunteer High" sensitivity uses U.S. census data on average earnings of 
individuals with bachelor's degrees as the cost of nonstudent volunteer time, and the cost 
of a paraeducator as the cost of student time. 
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dedicated to learning what works to improve the well-being of low-income people. 
Through its research and the active communication of its findings, MDRC seeks to 
enhance the effectiveness of social and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is 
best known for mounting rigorous, large-scale, real-world tests of new and existing 
policies and programs. Its projects are a mix of demonstrations (field tests of 
promising new program approaches) and evaluations of ongoing government and 
community initiatives. MDRC’s staff bring an unusual combination of research and 
organizational experience to their work, providing expertise on the latest in qualitative 
and quantitative methods and on program design, development, implementation, and 
management. MDRC seeks to learn not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s 
findings in the broader context of related research — in order to build knowledge 
about what works across the social and education policy fields. MDRC’s findings, 
lessons, and best practices are proactively shared with a broad audience in the policy 
and practitioner community as well as with the general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range 
of policy areas and target populations. Once known primarily for evaluations 
of state welfare-to-work programs, today MDRC is also studying public school 
reforms, employment programs for ex-offenders and people with disabilities, and 
programs to help low-income students succeed in college. MDRC’s projects are 
organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the 
United Kingdom, MDRC conducts its projects in partnership with national, state, and 
local governments, public school systems, community organizations, and numerous 
private philanthropies. 
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